research article comparing the state-of-the-art efficient stated choice designs...

Research ArticleComparing the State-of-the-Art Efficient Stated Choice DesignsBased on Empirical Analysis

Li Tang1 Xia Luo1 Yang Cheng2 Fei Yang1 and Bin Ran2

1 School of Transportation and Logistics Southwest Jiaotong University Chengdu 610031 China2Department of Civil and Environmental Engineering University of Wisconsin-Madison Madison WI 53706 USA

Correspondence should be addressed to Li Tang tanglitrafficgmailcom

Received 16 November 2013 Accepted 12 January 2014 Published 20 February 2014

Academic Editor Wuhong Wang

Copyright copy 2014 Li Tang et al This is an open access article distributed under the Creative Commons Attribution License whichpermits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

The stated choice (SC) experiment has been generally regarded as an effective method for behavior analysis Among all the SCexperimental design methods the orthogonal design has been most widely used since it is easy to understand and constructHowever in recent years a stream of research has put emphasis on the so-called efficient experimental designs rather than keepingthe orthogonality of the experiment as the former is capable of producing more efficient data in the sense that more reliableparameter estimates can be achieved with an equal or lower sample size This paper provides two state-of-the-art methods calledoptimal orthogonal choice (OOC) and 119863-efficient design More statistically efficient data is expected to be obtained by eithermaximizing attribute level differences or minimizing the119863-error a statistic corresponding to the asymptotic variance-covariance(AVC) matrix of the discrete choice model when using these two methods respectively Since comparison and validation in thefield of these methods are rarely seen an empirical study is presented 119863-error is chosen as the measure of efficiency The resultshows that both OOC and119863-efficient design are more efficient At last strength and weakness of orthogonal OOC and119863-efficientdesign are summarized

1 Introduction

Abundant and accurate data is the foundation of study Todate emerging technologies are largely introduced in datamining and processing [1] In [2 3] how to obtain high-quality data with the high-tech in the field of IntelligentTransportation System is deeply discussed However theimportance of refining data collection technique has notraised researchersrsquo attention in behavior analysis area untillast decades such as travel mode choice and safety in [4 5]The purpose of conducting choice experiments is to collectdata that can be used to estimate the independent influenceof attributes on observed choicesThere are two paradigms ofchoice data revealed preference (RP) and stated choice (SC)data Typically in RP surveys respondents are asked to recallinformation about hisher last choice including alternativesand attribute levels available in real market Differently SCexperiments present sampled respondents with a numberof different hypothetical choice situations each consistingof a universal but finite set of alternatives defined on a

number of attribute dimensions Thus SC data collectedon 300 respondents each of whom is asked to make 8choices produces a total of 2400 choice observations while RPexperiment on the same sample size of respondents collectsonly 300 observed choices

SC experiment has been widely used because it canobserve choices on alternatives which do not exist in thecurrent market So analysts are able to predict for examplethe share rate of a newly introduced transportation modeAnother reason of its popularity lies in the ability to providevariability in attributes in a relatively small sample sizecompared with RP experiment with which better estimationof influence of each attribute on choice can be achievedUsually respondents in a SC experiment will be faced withsome ldquoselectedrdquo choice situations considering that makingchoices among all the possible combinations of attributelevels is toomany to accomplish for a single respondentThushow analysts distribute the levels of the design attributes inan experiment plays a big role It may impact upon not onlywhether or not an independent assessment of contribution

Hindawi Publishing CorporationMathematical Problems in EngineeringVolume 2014 Article ID 740612 8 pageshttpdxdoiorg1011552014740612

2 Mathematical Problems in Engineering

of each attribute to the choices observed can be determinedbut also the ability of the experiment to detect statisticalrelationships that may exist within the data

Historically researchers have relied on orthogonal exper-imental designs in which the attributes of the experiment arestatistically independent by forcing them to be orthogonal[6] As such orthogonal designs theoretically allow for anindependent determination of each attributersquos influence uponthe observed choices To generate an orthogonal designusually but not necessarily the first step is to generate a fullfactorial design a design which contains all possible attributelevel combinationsMathematically a full factorial designwillproduceprod119896

119896=1119871119896 choice situations where 119871119896 is the number of

levels assigned to attribute 119896 (eg a design with 4 attributestwo with 2 attribute levels one with 3 levels and one with 4levels will produce a full factorial design with 2times2times3times4 = 48choice situations)The second step is to take a subset of choicesituations from the full factorial design which is known asfractional factorial designs Randomized cyclical Bayesianand fold over procedures are the common approaches used togenerate fractional factorial design in [7ndash10] It is noteworthythat in either a full factorial or a fractional factorial designorthogonality is kept between two random attributes Theonly difference between them is that orthogonality will not bekept in terms of interaction effects (ie the influence of twoor more attribute columns multiplied together) in fractionalfactorial design Here are some simple rules to check whethera fractional factorial design is orthogonal or not

(i) Every level of every attribute appears at same times

(ii) All possible attribute level combinations of randomtwo attributes appear at same times

While orthogonal design has long been used in practicein [11 12] there is a stream of researchers in recent yearsdoubted the importance of orthogonality in SC data whenit is used to estimate discrete choice model not to mentionwhether orthogonality can be kept in reality Orthogonalityis important in linear models since it avoids multicollinearityproblem and also minimized variance-covariance matrix ofthe estimated model in which way standard errors of param-eter estimates are also minimized Unfortunately discretechoice model is nonlinear thus the derivation of its parame-tersrsquo variance-covariancematrix is very different from thewayin linear models Seeing from that keeping orthogonality ofthe parameters has little to dowithminimizing their standarderrors

Acknowledgment of this fact has led researchers totransfer their efforts to obtain experimental designs thatmin-imize the asymptotic variance-covariance (AVC) matrix ofdiscrete choice models and provide more reliable parameterestimates with an equal or lower sample size Such designsare called efficient designs To date most research has beenfocused on developing methods to generate efficient designscomparison and validation of these methods in practice arerarely seenThis paper provides two state-of-the-art methodscalled optimal orthogonal choice (OOC) and 119863-efficientdesign Their performance on both theoretical efficiency and

Table 1 Optimal orthogonal choice design for 2 alternatives with 3binary attributes

Choice situation Alternative 1 Alternative 21198831

1198832

1198833

1198831

1198832

1198833

1 0 0 0 1 1 12 0 1 1 1 0 03 1 0 1 0 1 04 1 1 0 0 0 1

practical use are compared with the conventional orthogonaldesign

The remainder of this paper is organized as follows InSection 2 OOC and 119863-efficient design are introduced InSection 3 SC experiment design using orthogonal OOCand119863-efficient methods separately is generatedThe result interms of 119863-errors are presented in Section 4 along with thesummarized strength and weakness

2 Efficient Experimental Design Methods

21 Optimal Orthogonal Choice Design Considering thepopularity and convenience of orthogonal design for analyz-ers in practice there is a stream of researchers in [13 14] whokept on exploring improved SC design method maintainingorthogonality which is called optimal orthogonal choicedesign The essential idea of OOC design is to maximize thedifferences of attribute levels across alternatives so that theparameters can be estimated in the largest extent of variety ofattribute levels as well as independently The basic process ofgenerating an OOC design is as follows

Step 1 Generate a fractional factorial orthogonal design foralternative 1119873 represents the number of choice situations ofthe design

Step 2 Choose some systematic changes to get the allocationof attribute levels in alternative 2 from alternative 1 System-atic changes are certain rules to decide how the attributelevels change from alternative 1 and will be discussed in latercontext

Step 3 Choose another systematic change to get the alloca-tion of attribute levels in alternative 3 from alternative 1

Step 4 Keep doing this until all the alternatives are deter-mined

It will be much easier to understand this method bystartingwith a binary attribute level design Again we assumethat 119871119896 is the number of levels assigned to generic attribute 119896for alternative 119895 represented by 0 1 119871119896 minus 1 In a designfor 2 alternatives and 3 attributes each with 2 levels anorthogonal design in 4 choice situations for alternative 1can be firstly generated Then 0rsquos and 1rsquos in alternative 1 areinterchanged in alternative 2Thus the attribute levels of eachattribute are forced to be different across alternatives Theresult is shown in Table 1

Mathematical Problems in Engineering 3


Choice situation Alternative 1 Alternative 2 Alternative 31198831

1198832

1198833

1198831

1198832

1198833

1198831

1198832

1198833

1 0 0 0 1 1 0 0 0 12 0 1 1 1 0 1 0 1 03 1 0 1 0 1 1 1 0 04 1 1 0 0 0 0 1 1 1

To generate OOC design for more alternatives it isnecessary to introduce 119878119896 to represent the largest number ofdifferent pairs appeared between alternatives for a specificattribute The equation of 119878119896 is shown as follows where 119869stands for the number of alternatives in choice set

119878119896 =

(1198692minus 1)

4 119897119896 = 2 119869 odd

1198692

4 119897119896 = 2 119869 even

(1198692minus (119897119896119909

2+ 2119909119910 + 119910))

2 2 lt 119897119896 le 119869

119869 (119869 minus 1)

2 119897119896 ge 119869

(1)

In an example of OOC design for 3 alternatives and 3attributes each with 2 levels shown in Table 1 we can get119878119896 = (119869

2minus 1)4 = 2 for all the attributes For instance in the

first choice situation (000 110 and 001) the attribute levelsdiffer twice for each attribute (ie for attribute 1198831 the levelsare 010 creating 3 pairs (01 10 and 00) in which two of them(01 10) are different)

We can see from Table 1 that the distribution of attributelevels in alternative 2 is obtained by interchanging 0rsquos and1rsquos in 1198831 and 1198832 in alternative 1 and that in alternative 3 isobtained by interchanging 0rsquos and 1rsquos in 1198833 in alternative 1These systematic changes can be also described as addinga generator in alternative 1 to get alternative 2 and addinganother generator to get alternative 3 The addition is per-formed in modulo arithmetic according to the number oflevels for a specific attribute Here in the example 119871119896 = 2

for all attributes thus when a generator 110 is added to thechoice situations in alternative 1 in modulo 2 arithmetic likethis 000 + 110 equiv 110 011 + 110 equiv 101 and so on alternative2 is obtained Alternative 3 is generated in the same wayby adding a generator 001 to alternative 1 Notice that thegenerators added to alternatives must have a value of 119878119896 = 2

(ie generators (000110 and 001) used in Table 2 can meetthe requirement while another generators (000 100 and 010)cannot not)

Designs for any choice set size with any number ofattributes each having any number of levels can be generatedin similar way However a big limitation of OOC design isthat it can only generate designs for generic attributes accord-ing to the principle of this method How alternative-specificattributes distribute across alternatives is rarely discussed

in the literature Another shortcoming of OOC design isthat it may produce a lot of unreasonable combinationsby forcing maximum level differences in attributes acrossalternatives Answers from responders who have to makedecision in such choice situations may not reflect their actualchoice statements since the ldquoproper alternativerdquo may not becontained in the questionnaire

22 119863-Efficient Design While people who raise OOCmethod keep on improving design with remaining orthog-onality another stream of researchers goes straight forwardto increasing the statistical efficiency of the design by min-imizing the elements of the asymptotic variance-covariance(AVC)matrix of discrete choice modelsThe AVCmatrix canbe obtained by taking the negative inverse of the expectedsecond derivatives of the log-likelihood function of themodelproofed in [15] To interpret the process of calculating theAVC matrix and the measure of efficiency of a design herewe briefly introduce the most well-known multinomial logitmodel

Assume an individual facedwith alternative 119895 = 1 2 119869in choice situation 119899 = 1 2 119873 The utility of anindividual for alternative 119895 in choice situation 119899 can beexpressed as

119880119895119899 = 119881119895119899 + 120576119895119899 (2)

where119881119895119899 represents observed part of utility for each alterna-tive 119895 in choice situation 119899 It is assumed to be a linear additivefunction of several attributes with corresponding weightsThese weights are unknown parameters to be estimatedand can be divided into two categories generic parametersand alternative-specific parameters The generic parametersand alternative-specific parameters can be denoted by 120573lowast

119896

119896 = 1 119870lowast and 120573119895119896 119896 = 1 119870119895 respectively

with their associated attribute levels 119909lowast119895119896119899

and 119909119895119896119899 for eachchoice situation 119899 Thus the total number of parametersto be estimated equal to 119870 = 119870

lowast+ sum119869

119895=1119870119895 119881119895119899 can be

expressed as

119881119895119899 =

119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899

+

119870119895

sum

119896=1

120573119895119896119909119895119896119899 forall 119895 = 1 119869 forall 119899 = 1 119873

(3)

where 120576119895119899 is the unobserved component which is indepen-dently and identically extreme value type one distributedThe probability 119875119895119899 that an individual chooses alternative 119895in choice situation 119899 becomes

119875119895119899 =

exp (119881119895119899)

sum119869

119895=1exp (119881119895119899)

forall119895 = 1 119869 forall119899 = 1 119873 (4)


Considering that the most popular way to estimateparameters is maximum likelihood estimation the log-likelihood function of parameters for a single respondent canbe expressed as

119871 (120573lowast 120573) =

119873

sum

119899=1

119869

sum

119895=1

119910119895119899 log119875119895119899

=

119873

sum

119899=1

[

[

119869

sum

119895=1

119910119895119899(

119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899+

119870119895

sum

119896=1

120573119895119896119909119895119896119899)

minus log(119869

sum

119894=1

exp(119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899+

119870119895

sum

119896=1

120573119895119896119909119895119896119899))]

]

(5)

where 119910 represents the binary outcome of all choice situa-tions While alternative 119895 is chosen in choice situation 119899 119910119895119899equals one otherwise it is zero Then the AVC matrix canbe expressed as the second derivative of the log-likelihoodfunction as follows

1205972119871 (120573lowast 120573)

120597120573lowast

1198961

120597120573lowast

1198962

= minus

119873

sum

119899=1

119869

sum

119895=1

119909lowast

1198951198961119899119875119895119899(119909

lowast

1198951198961119899minus

119869

sum

119894=1

119909lowast

1198941198962119899119875119894119899)

forall1198961 1198962 = 1 119870lowast

(6)

1205972119871 (120573lowast 120573)

12059712057311989511198961

120597120573lowast

1198962

= minus

119873

sum

119899=1

119909119895111989611198991198751198951119899(119909lowast

11989511198962119899minus

119869

sum

119894=1

119909lowast

1198941198962119899119875119894119899)

forall 1198951 = 1 119869 1198961 = 1 1198701198951

1198962 = 1 119870lowast

(7)

1205972119871 (120573lowast 120573)

12059712057311989511198961

12059712057311989521198962

=

119873

sum

119899=1

119909119895111989611198991199091198952119896211989911987511989511198991198751198952119899 if 1198951 = 1198952

minus

119873

sum

119899=1

11990911989511198961119899119909119895211989621198991198751198951119899 (1 minus 119875119895

2119899) if 1198951 = 1198952

forall1198951 = 1 119869 119896119894 = 1 119870119895119894

(8)

Equations (6)ndash(8) represent functions that allow genericand alternative-specific parameters In the case where onlygeneric parameters exist only (6) remains and when thereare only alternative-specific parameters (8) remains Inaddition if there are 119872 identical respondents these secondderivatives are multiplied by119872

Let (120573lowast

120573) denote the true values of the parameters TheFisher information matrix 119868 is defined as the expected valuesof the second derivative of the log-likelihood function

119868 (120573lowast

120573) = 119872 sdot

1205972119871 (120573lowast

120573)

1205971205731205971205731015840 (9)

Hence the AVC matrix can be expressed as a 119870 times 119870 matrixthat is equal to the negative inverse of the Fisher informationmatrix [16]

Ω = minus[119868 (120573lowast

120573)]

minus1

= minus1

119872

[

[

1205972119871 (120573lowast

120573)

1205971205731205971205731015840]

]

minus1

(10)

Rather than working with each element within the AVCmatrix directly a preferredmeasurewithin the literature is119863-error calculated by taking the determinant of the AVCmatrixand scaling this value by the number of parameters 119870 It iscommon to assume that a single respondent (ie 119872 = 1)represents all respondents an assumption consistent with themultinomial logit (MNL) model form [8 17] Designs thataim at minimizing119863-error are called119863-efficient designs

Since the calculation of 119863-error involves the values ofparameters approaches to determine 119863-error have beenimproved in recent years In our empirical study later weuse 119863119901-error as the statistic to measure the efficiency ofexperimental designs To calculate 119863119901-error nonzero priorsare needed It can be expressed as follows

119863119901-error = minus[

[

det1205972119871 (120573lowast

120573)

1205971205731205971205731015840]

]

minus1119870

(11)

According to the 119863-efficient method we can figure outthat for a given sample size attempts to minimize 119863-error statistic will directly lead to the minimization of theAVC matrix Meanwhile by taking the square root of thediagonal elements (including 119872) of the AVC matrix theminimization of asymptotic standard errors is achievedThusthe asymptotic standard error of the parameter estimateswill diminish in terms of statistical significance from eachadditional respondent added to a survey

Figure 1(a) reveals that the standard error decreases byincreasing sample size at the beginning of a given design119883

I Exceeding a certain limit enlarging sample size haslittle impact on demising standard error On the other sideFigure 1(b) shows that investing a more efficient design 119883II

can lead to larger decreases in standard error In other wordsa smaller sample size may be satisfied at a certain levelof standard error when using efficient designs rather thancommon ones

3 Empirical Study

In this section three experimental designs which useddifferent methods (orthogonal OOC and 119863-efficient) aregenerated They will be used to obtain trip mode choicedata on a corridor connecting two large business districtsin Chengdu Shawan conference and exhibition center andJinsha station are chosen as the origin and destination forthe survey Three alternatives are involved car taxi and busA typical multinomial logit model in transportation will beformulated and serve as the basis of most of the analyses in


N

se1 (XI 120573)

seN (XI 120573)

0 10 20 30 40 50

Stan

dard

erro

r

(a) Larger samples

Stan

dard

erro

r

se1 (XI 120573)

seN (XI 120573)

seN (XII 120573)

N

0 10 20 30 40 50

(b) Better designs

Figure 1 Comparison of investing in larger sample sizes versusmore efficient designs

the subsequent section The observed part of utility of everyalternative is expressed as follows

119881car= 120573

car0+ 1205731TT

car+ 1205732TC

car

119881taxi

= 120573taxi0

+ 1205731TTtaxi

+ 1205732TCtaxi

119881bus

= 1205731TTbus

+ 1205732TCbus

(12)

where TT represents travel time TC represents travel costFor car users119879119862 equals the fuel cost For taxi users TCequalsthe money paid for the trip For bus users TC equals theticket price Seeing from (12) parameters for TT and TC aregeneric across three alternatives Thus four parameters aregoing to be estimated in total (two of them are alternative-specific constant which has nothing to dowith any attribute)The attribute levels and prior information about parametersare given in Table 3 based on previous study results as wellas to preserve realistic estimates for the private and publictransport alternatives

In order to obtain better estimation of parameters threelevels are set for each attribute for maximum variation as

Table 3 Prior parameter values and attribute levels for case study

(a)

Prior parameter values120573car0

120573taxi0

1205731 1205732

1 minus06 minus045 minus09

(b)

Attribute levelCar Taxi Bus

TTcar TCcar TTtaxi TCtaxi TTbus TCbus

(min) (RMB) (min) (RMB) (min) (RMB)15 4 15 16 25 120 5 20 18 30 230 7 30 22 38 3

much as possible The values of TTcar TTtaxi and TTbus aremeasured in free normal and congested traffic flow Thevalues of TCcar are calculated as the kilometers betweenthe origin and destination by the consumed oil price under73 RMBliter 77 RMBliter and 8 RMBliter The values ofTCtaxi are measured in free normal and congested trafficflow The value of TCbus is based on the current price andplusminus 1 RMB The number of choice situations (ie 18)is selected such that attribute level balance can be achievedObviously this number is too large for a single respondentThus we introduce a block variable to divide the design intosmaller parts (ie here we block the design into three parts sothat six choice situations are provided to a single respondent)Each block is not orthogonal by itself but in combinationwith other blocks Attribute level balance is maintained asmuch as possible in each block

We generate three different (attribute level balanced)designs with 18 choice situations assuming the above MNLmodel using the software Ngene 111 The design results areshown in Table 4 as well as119863-error value for each design

4 Results and Discussion

As expected the two efficient designs produce lower 119863-error value (0118313 and 0114612 for the OOC design and119863-efficient design resp) while orthogonal design produceshigher 119863-error value (0194724) seeing from Table 4 The119863-error of the orthogonal design is 164 times greater thanthe 119863-error value of the OOC design and 169 times greaterthan 119863-efficient design This suggests that on average theasymptotic standard errors of the parameter estimates usingthe orthogonal design will be 128 to 131 times larger thanthe efficient designs Clearly the efficient designs are able toprovide more reliable parameter estimates than orthogonaldesign

On the other hand comparing the two efficient designs119863-efficient outperforms OOC in terms of the 119863-error value(0114612 versus 0118313) rsquoFurthermore since the OOCmethod can only generate designs with generic attributes theuse of it is largely limited In a word with high statistical


Table 4 Experimental designs for empirical study

Choice situation Car Taxi Bus BlockTTcar (min) TCcar (RMB) TTtaxi (min) TCtaxi (RMB) TTbus (min) TCbus (RMB)

Orthogonal design for MNL model (119863-error = 0194724)1 15 4 15 16 25 1 12 20 5 20 18 30 2 13 30 7 30 22 38 3 14 15 5 20 22 38 1 15 20 7 30 16 25 2 16 30 4 15 18 30 3 17 30 7 20 18 25 1 28 15 4 30 22 30 2 29 20 5 15 16 38 3 210 20 7 15 22 30 1 211 30 4 20 16 38 2 212 15 5 30 18 25 3 213 30 5 30 16 30 1 314 15 7 15 18 38 2 315 20 4 20 22 25 3 316 20 4 30 18 38 1 317 30 5 15 22 25 2 318 15 7 20 16 30 3 3

Orthogonal optimal design for MNL model (119863-error = 0118313)1 15 4 20 18 38 3 12 30 7 15 16 30 2 13 15 5 20 22 38 1 14 20 4 30 18 25 3 15 30 5 15 22 30 1 16 20 7 30 16 25 2 17 20 7 30 16 25 2 28 30 5 15 22 30 1 29 15 4 20 18 38 3 210 20 4 30 18 25 3 211 15 5 20 22 38 1 212 30 7 15 16 30 2 213 15 7 20 16 38 2 314 20 5 30 22 25 1 315 30 4 15 18 30 3 316 30 4 15 18 30 3 317 20 5 30 22 25 1 318 15 7 20 16 38 2 3

119863-efficient design for MNL model (119863-error = 0114612)1 30 5 20 18 25 1 12 20 7 15 16 38 2 13 30 4 15 22 25 3 14 15 5 30 16 30 3 15 15 5 30 16 38 2 16 30 5 20 22 25 1 17 20 4 15 22 30 2 28 15 7 30 22 38 1 29 15 7 20 22 38 1 210 30 5 15 16 30 3 2


Table 4 Continued


11 15 5 30 18 38 2 212 20 4 20 18 25 3 213 20 7 30 16 25 1 314 20 4 20 18 30 2 315 30 7 15 16 30 3 316 15 7 30 18 38 1 317 20 4 20 18 30 3 318 30 4 15 22 25 2 3

Table 5 Comparison of orthogonal OOC and 119863-efficient design

Method Advantage Disadvantage

Orthogonal

(i) It is the most widely used method andeasy to construct or obtain(ii) There are no correlations betweenattribute levels thus it allows for anindependent estimation of the influence ofeach attribute on choice

(i) There are too many choicesituationsquestions for a single respondent(ii) Orthogonally it is hard to maintain inactual design subsets replicated unevenlyintroducing sociodemographic variable andallocation bias of the implausible choicesituation(iii) It may contain ldquouselessrdquo choice situations

Optimal orthogonal choice(i) Attribute level differences are maximized(ii) Choice situations will be reduced as wellas attaining the designrsquos orthogonality

(i) It can only generate designs for genericattributes the rules for setting upalternative-specific attributes are not clear rightnow(ii) Unreasonable combinations of attributelevels may appear thus the ldquoreal choicerdquo ofrespondents is hard to capture

119863-efficient

(i) The smaller the asymptotic standarderrors achieved the smaller the width of theconfidence intervals observed around theparameters estimates will be(ii) 119905-Radios will be maximized thusproducing more reliable study results andanalyst is able to minimize the sample size

(i) In general not orthogonal (not thatimportant)(ii) Advanced knowledge of the parameterestimates is needed(iii) It needs more computation power

efficiency and wide applicability 119863-efficient design achievesthe best performance

Furthermost research focused ondeveloping one of theseexperimental design methods by far Though there may be afew of discussions about strength and shortcomings of everymethod separately comparisons are rarely found in theory orpractice area [12 18 19] Here we conclude the advantagesand disadvantages of these three methods in Table 5 Also apopularity rate is given as a reference of their applications inthe field

5 Conclusion

TheSCexperiment has been generally regarded as an effectivemethod for discrete choice analysis especially for newlyintroduced alternatives The high cost on survey forcesresearchers to find more efficient design methods to obtainbetter estimation on parameters instead of investing a largersample size Though orthogonal design has been used as the

major experimental design method orthogonality is not thatimportant in the nonlinear discrete choice models In thispaper we provide two state-of-the-art efficient designs OOCand119863-efficient design By comparing orthogonal OOC and119863-efficient design in both theory and practice we find thatefficient designs aremore capable of producingmore efficientdata in the sense that more reliable parameter estimates canbe achieved with an equal or lower sample size The gen-eration process requires the assumption of prior parameterestimates and model structure to construct AVC matrix Theresult suggests a move away from orthogonal designs for SCexperiments towards119863-efficient designs whichmake relativediscrete choice models being more fit with such data

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper


Acknowledgments

This study is supported by the 2014 Doctoral InnovationFunds of Southwest Jiaotong University the FundamentalResearch Funds for the Central Universities (noA0920502051307-03) Specialized Research Fund for theDoctoral ProgramofHigher Education (no 20130184110020)National Natural Science Foundation of China (no51178403) and Sichuan Province Science and TechnologySupport Project (no 2011FZ0050)

References

[1] H Tan G Feng J Feng W Wang Y Zhang and F LildquoA tensor-based method for missing traffic data completionrdquoTransportation Research Part C vol 28 pp 15ndash27 2013

[2] H Tan J Feng G Feng W Wang and Y-J Zhang ldquoTrafficvolume data outlier recovery via tensor modelrdquo MathematicalProblems in Engineering Article ID 164810 8 pages 2013

[3] H Tana B Chenga J Fenga G Fenga W Wanga and Y JZhangb ldquoLow-n-rank tensor recovery based on multi-linearaugmented lagrange multiplier methodrdquo Neurocomputing vol119 pp 144ndash152

[4] WWangW Zhang H GuoH Bubb andK Ikeuchi ldquoA safety-based approaching behavioural model with various drivingcharacteristicsrdquo Transportation Research C vol 19 no 6 pp1202ndash1214 2011

[5] WWang X Jiang S Xia and Q Cao ldquoIncident tree model andincident tree analysis method for quantified risk assessment anin-depth accident study in traffic operationrdquo Safety Science vol48 no 10 pp 1248ndash1262 2010

[6] J J Louviere D A Hensher and J D Swait Stated ChoiceMeth-ods Analysis and Applications Cambridge University PressCambridge UK 2000

[7] P Galilea and J D D Ortuzar ldquoValuing noise level reductionsin a residential location contextrdquoTransportationResearchD vol10 no 4 pp 305ndash322 2005

[8] D S Bunch J J Louviere and D Anderson ldquoA comparisonof Experimental Design Strategies for Choice-based ConjointAnalysis with Generic-attribute Multinomial Logit ModelsrdquoWorking Paper Graduate School of Management University ofCalifornia Rickey Davis Ala USA 1994

[9] J J Louviere and GWoodworth ldquoDesign and analysis of simu-lated consumer choice or allocation experiments an approachbased on aggregate datardquo Journal of Marketing Research vol 20pp 350ndash367 1983

[10] J Huber and K Zwerina ldquoThe importance of utility balance inefficient choice designsrdquo Journal of Marketing Research vol 33no 3 pp 307ndash317 1996

[11] J M Rose and M C J Bliemer ldquoConstructing efficient statedchoice experimental designrdquo ITLS Working Paper ITLS-WP-05-07 2005

[12] M C J Bliemer J M Rose and D A Hensher ldquoEfficient statedchoice experiments for estimating nested logit modelsrdquo Trans-portation Research B vol 43 no 1 pp 19ndash35 2009

[13] D J Street and L Burgess ldquoOptimal and near-optimal pairs forthe estimation of effects in 2-level choice experimentsrdquo Journalof Statistical Planning and Inference vol 118 no 1-2 pp 185ndash1992004

[14] L Burgess and D J Street ldquoOptimal designs for choiceexperiments with asymmetric attributesrdquo Journal of StatisticalPlanning and Inference vol 134 no 1 pp 288ndash301 2005

[15] D McFadden ldquoConditional logit analysis of qualitative choicebehaviorrdquo in Frontiers in Econometrics P Zarembka Ed pp105ndash142 Academic Press New York NY USA 1974

[16] DMcFadden ldquoThe choice theory approach tomarket researchrdquoMarketing Science vol 5 no 4 pp 275ndash297 1986

[17] J M Rose and M C J Bliemer ldquoStated preference experi-mental design strategiesrdquo in Transport Modeling Handbooks inTransport D A Hensher and K Button Eds vol 1 chapter 8Elsevier Science Oxford UK 2nd edition 2007

[18] J M Rose M C J Bliemer D A Hensher and A T CollinsldquoDesigning efficient stated choice experiments in the presenceof reference alternativesrdquo Transportation Research B vol 42 no4 pp 395ndash406 2008

[19] D J Street L Burgess and J J Louviere ldquoQuick and easy choicesets constructing optimal and nearly optimal stated choiceexperimentsrdquo International Journal of Research in Marketingvol 22 no 4 pp 459ndash470 2005

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

MathematicsJournal of


Mathematical Problems in Engineering

Hindawi Publishing Corporationhttpwwwhindawicom

Differential EquationsInternational Journal of

Volume 2014

Applied MathematicsJournal of


Probability and StatisticsHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Journal of


Mathematical PhysicsAdvances in

Complex AnalysisJournal of


OptimizationJournal of


CombinatoricsHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of


Operations ResearchAdvances in

Journal of


Function Spaces

Abstract and Applied AnalysisHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of Mathematics and Mathematical Sciences


The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014


Algebra

Discrete Dynamics in Nature and Society



Decision SciencesAdvances in

Discrete MathematicsJournal of


Volume 2014 Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Stochastic AnalysisInternational Journal of


of each attribute to the choices observed can be determinedbut also the ability of the experiment to detect statisticalrelationships that may exist within the data

Historically researchers have relied on orthogonal exper-imental designs in which the attributes of the experiment arestatistically independent by forcing them to be orthogonal[6] As such orthogonal designs theoretically allow for anindependent determination of each attributersquos influence uponthe observed choices To generate an orthogonal designusually but not necessarily the first step is to generate a fullfactorial design a design which contains all possible attributelevel combinationsMathematically a full factorial designwillproduceprod119896

119896=1119871119896 choice situations where 119871119896 is the number of

levels assigned to attribute 119896 (eg a design with 4 attributestwo with 2 attribute levels one with 3 levels and one with 4levels will produce a full factorial design with 2times2times3times4 = 48choice situations)The second step is to take a subset of choicesituations from the full factorial design which is known asfractional factorial designs Randomized cyclical Bayesianand fold over procedures are the common approaches used togenerate fractional factorial design in [7ndash10] It is noteworthythat in either a full factorial or a fractional factorial designorthogonality is kept between two random attributes Theonly difference between them is that orthogonality will not bekept in terms of interaction effects (ie the influence of twoor more attribute columns multiplied together) in fractionalfactorial design Here are some simple rules to check whethera fractional factorial design is orthogonal or not

(i) Every level of every attribute appears at same times

(ii) All possible attribute level combinations of randomtwo attributes appear at same times

While orthogonal design has long been used in practicein [11 12] there is a stream of researchers in recent yearsdoubted the importance of orthogonality in SC data whenit is used to estimate discrete choice model not to mentionwhether orthogonality can be kept in reality Orthogonalityis important in linear models since it avoids multicollinearityproblem and also minimized variance-covariance matrix ofthe estimated model in which way standard errors of param-eter estimates are also minimized Unfortunately discretechoice model is nonlinear thus the derivation of its parame-tersrsquo variance-covariancematrix is very different from thewayin linear models Seeing from that keeping orthogonality ofthe parameters has little to dowithminimizing their standarderrors

Acknowledgment of this fact has led researchers totransfer their efforts to obtain experimental designs thatmin-imize the asymptotic variance-covariance (AVC) matrix ofdiscrete choice models and provide more reliable parameterestimates with an equal or lower sample size Such designsare called efficient designs To date most research has beenfocused on developing methods to generate efficient designscomparison and validation of these methods in practice arerarely seenThis paper provides two state-of-the-art methodscalled optimal orthogonal choice (OOC) and 119863-efficientdesign Their performance on both theoretical efficiency and


Choice situation Alternative 1 Alternative 21198831

1198832

1198833

1198831

1198832

1198833

1 0 0 0 1 1 12 0 1 1 1 0 03 1 0 1 0 1 04 1 1 0 0 0 1

practical use are compared with the conventional orthogonaldesign

The remainder of this paper is organized as follows InSection 2 OOC and 119863-efficient design are introduced InSection 3 SC experiment design using orthogonal OOCand119863-efficient methods separately is generatedThe result interms of 119863-errors are presented in Section 4 along with thesummarized strength and weakness

2 Efficient Experimental Design Methods

21 Optimal Orthogonal Choice Design Considering thepopularity and convenience of orthogonal design for analyz-ers in practice there is a stream of researchers in [13 14] whokept on exploring improved SC design method maintainingorthogonality which is called optimal orthogonal choicedesign The essential idea of OOC design is to maximize thedifferences of attribute levels across alternatives so that theparameters can be estimated in the largest extent of variety ofattribute levels as well as independently The basic process ofgenerating an OOC design is as follows

Step 1 Generate a fractional factorial orthogonal design foralternative 1119873 represents the number of choice situations ofthe design

Step 2 Choose some systematic changes to get the allocationof attribute levels in alternative 2 from alternative 1 System-atic changes are certain rules to decide how the attributelevels change from alternative 1 and will be discussed in latercontext

Step 3 Choose another systematic change to get the alloca-tion of attribute levels in alternative 3 from alternative 1

Step 4 Keep doing this until all the alternatives are deter-mined

It will be much easier to understand this method bystartingwith a binary attribute level design Again we assumethat 119871119896 is the number of levels assigned to generic attribute 119896for alternative 119895 represented by 0 1 119871119896 minus 1 In a designfor 2 alternatives and 3 attributes each with 2 levels anorthogonal design in 4 choice situations for alternative 1can be firstly generated Then 0rsquos and 1rsquos in alternative 1 areinterchanged in alternative 2Thus the attribute levels of eachattribute are forced to be different across alternatives Theresult is shown in Table 1




1198832

1198833

1198831

1198832

1198833

1198831

1198832

1198833

1 0 0 0 1 1 0 0 0 12 0 1 1 1 0 1 0 1 03 1 0 1 0 1 1 1 0 04 1 1 0 0 0 0 1 1 1


119878119896 =

(1198692minus 1)

4 119897119896 = 2 119869 odd

1198692

4 119897119896 = 2 119869 even

(1198692minus (119897119896119909

2+ 2119909119910 + 119910))

2 2 lt 119897119896 le 119869

119869 (119869 minus 1)

2 119897119896 ge 119869

(1)











119880119895119899 = 119881119895119899 + 120576119895119899 (2)


119896




lowast+ sum119869

119895=1119870119895 119881119895119899 can be

expressed as

119881119895119899 =

119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899

+

119870119895

sum

119896=1

120573119895119896119909119895119896119899 forall 119895 = 1 119869 forall 119899 = 1 119873

(3)


119875119895119899 =

exp (119881119895119899)

sum119869

119895=1exp (119881119895119899)

forall119895 = 1 119869 forall119899 = 1 119873 (4)



119871 (120573lowast 120573) =

119873

sum

119899=1

119869

sum

119895=1

119910119895119899 log119875119895119899

=

119873

sum

119899=1

[

[

119869

sum

119895=1

119910119895119899(

119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899+

119870119895

sum

119896=1

120573119895119896119909119895119896119899)

minus log(119869

sum

119894=1

exp(119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899+

119870119895

sum

119896=1

120573119895119896119909119895119896119899))]

]

(5)


1205972119871 (120573lowast 120573)

120597120573lowast

1198961

120597120573lowast

1198962

= minus

119873

sum

119899=1

119869

sum

119895=1

119909lowast

1198951198961119899119875119895119899(119909

lowast

1198951198961119899minus

119869

sum

119894=1

119909lowast

1198941198962119899119875119894119899)

forall1198961 1198962 = 1 119870lowast

(6)

1205972119871 (120573lowast 120573)

12059712057311989511198961

120597120573lowast

1198962

= minus

119873

sum

119899=1

119909119895111989611198991198751198951119899(119909lowast

11989511198962119899minus

119869

sum

119894=1

119909lowast

1198941198962119899119875119894119899)

forall 1198951 = 1 119869 1198961 = 1 1198701198951

1198962 = 1 119870lowast

(7)

1205972119871 (120573lowast 120573)

12059712057311989511198961

12059712057311989521198962

=

119873

sum

119899=1

119909119895111989611198991199091198952119896211989911987511989511198991198751198952119899 if 1198951 = 1198952

minus

119873

sum

119899=1

11990911989511198961119899119909119895211989621198991198751198951119899 (1 minus 119875119895

2119899) if 1198951 = 1198952

forall1198951 = 1 119869 119896119894 = 1 119870119895119894

(8)


Let (120573lowast


119868 (120573lowast

120573) = 119872 sdot

1205972119871 (120573lowast

120573)

1205971205731205971205731015840 (9)



120573)]

minus1

= minus1

119872

[

[

1205972119871 (120573lowast

120573)

1205971205731205971205731015840]

]

minus1

(10)



119863119901-error = minus[

[

det1205972119871 (120573lowast

120573)

1205971205731205971205731015840]

]

minus1119870

(11)





3 Empirical Study



N

se1 (XI 120573)

seN (XI 120573)

0 10 20 30 40 50

Stan

dard

erro

r

(a) Larger samples

Stan

dard

erro

r

se1 (XI 120573)

seN (XI 120573)

seN (XII 120573)

N

0 10 20 30 40 50

(b) Better designs



119881car= 120573

car0+ 1205731TT

car+ 1205732TC

car

119881taxi

= 120573taxi0

+ 1205731TTtaxi

+ 1205732TCtaxi

119881bus

= 1205731TTbus

+ 1205732TCbus

(12)




(a)


120573taxi0

1205731 1205732


(b)
















Table 4 Continued


11 15 5 30 18 38 2 212 20 4 20 18 25 3 213 20 7 30 16 25 1 314 20 4 20 18 30 2 315 30 7 15 16 30 3 316 15 7 30 18 38 1 317 20 4 20 18 30 3 318 30 4 15 22 25 2 3



Orthogonal





119863-efficient





5 Conclusion






Acknowledgments


References



























Volume 2014




Journal of











Journal of


Function Spaces






Algebra












1198832

1198833

1198831

1198832

1198833

1198831

1198832

1198833

1 0 0 0 1 1 0 0 0 12 0 1 1 1 0 1 0 1 03 1 0 1 0 1 1 1 0 04 1 1 0 0 0 0 1 1 1


119878119896 =

(1198692minus 1)

4 119897119896 = 2 119869 odd

1198692

4 119897119896 = 2 119869 even

(1198692minus (119897119896119909

2+ 2119909119910 + 119910))

2 2 lt 119897119896 le 119869

119869 (119869 minus 1)

2 119897119896 ge 119869

(1)











119880119895119899 = 119881119895119899 + 120576119895119899 (2)


119896




lowast+ sum119869

119895=1119870119895 119881119895119899 can be

expressed as

119881119895119899 =

119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899

+

119870119895

sum

119896=1

120573119895119896119909119895119896119899 forall 119895 = 1 119869 forall 119899 = 1 119873

(3)


119875119895119899 =

exp (119881119895119899)

sum119869

119895=1exp (119881119895119899)

forall119895 = 1 119869 forall119899 = 1 119873 (4)



119871 (120573lowast 120573) =

119873

sum

119899=1

119869

sum

119895=1

119910119895119899 log119875119895119899

=

119873

sum

119899=1

[

[

119869

sum

119895=1

119910119895119899(

119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899+

119870119895

sum

119896=1

120573119895119896119909119895119896119899)

minus log(119869

sum

119894=1

exp(119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899+

119870119895

sum

119896=1

120573119895119896119909119895119896119899))]

]

(5)


1205972119871 (120573lowast 120573)

120597120573lowast

1198961

120597120573lowast

1198962

= minus

119873

sum

119899=1

119869

sum

119895=1

119909lowast

1198951198961119899119875119895119899(119909

lowast

1198951198961119899minus

119869

sum

119894=1

119909lowast

1198941198962119899119875119894119899)

forall1198961 1198962 = 1 119870lowast

(6)

1205972119871 (120573lowast 120573)

12059712057311989511198961

120597120573lowast

1198962

= minus

119873

sum

119899=1

119909119895111989611198991198751198951119899(119909lowast

11989511198962119899minus

119869

sum

119894=1

119909lowast

1198941198962119899119875119894119899)

forall 1198951 = 1 119869 1198961 = 1 1198701198951

1198962 = 1 119870lowast

(7)

1205972119871 (120573lowast 120573)

12059712057311989511198961

12059712057311989521198962

=

119873

sum

119899=1

119909119895111989611198991199091198952119896211989911987511989511198991198751198952119899 if 1198951 = 1198952

minus

119873

sum

119899=1

11990911989511198961119899119909119895211989621198991198751198951119899 (1 minus 119875119895

2119899) if 1198951 = 1198952

forall1198951 = 1 119869 119896119894 = 1 119870119895119894

(8)


Let (120573lowast


119868 (120573lowast

120573) = 119872 sdot

1205972119871 (120573lowast

120573)

1205971205731205971205731015840 (9)



120573)]

minus1

= minus1

119872

[

[

1205972119871 (120573lowast

120573)

1205971205731205971205731015840]

]

minus1

(10)



119863119901-error = minus[

[

det1205972119871 (120573lowast

120573)

1205971205731205971205731015840]

]

minus1119870

(11)





3 Empirical Study



N

se1 (XI 120573)

seN (XI 120573)

0 10 20 30 40 50

Stan

dard

erro

r

(a) Larger samples

Stan

dard

erro

r

se1 (XI 120573)

seN (XI 120573)

seN (XII 120573)

N

0 10 20 30 40 50

(b) Better designs



119881car= 120573

car0+ 1205731TT

car+ 1205732TC

car

119881taxi

= 120573taxi0

+ 1205731TTtaxi

+ 1205732TCtaxi

119881bus

= 1205731TTbus

+ 1205732TCbus

(12)




(a)


120573taxi0

1205731 1205732


(b)
















Table 4 Continued


11 15 5 30 18 38 2 212 20 4 20 18 25 3 213 20 7 30 16 25 1 314 20 4 20 18 30 2 315 30 7 15 16 30 3 316 15 7 30 18 38 1 317 20 4 20 18 30 3 318 30 4 15 22 25 2 3



Orthogonal





119863-efficient





5 Conclusion






Acknowledgments


References



























Volume 2014




Journal of











Journal of


Function Spaces






Algebra











119871 (120573lowast 120573) =

119873

sum

119899=1

119869

sum

119895=1

119910119895119899 log119875119895119899

=

119873

sum

119899=1

[

[

119869

sum

119895=1

119910119895119899(

119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899+

119870119895

sum

119896=1

120573119895119896119909119895119896119899)

minus log(119869

sum

119894=1

exp(119870lowast

sum

119896=1

120573lowast

119896119909lowast

119895119896119899+

119870119895

sum

119896=1

120573119895119896119909119895119896119899))]

]

(5)


1205972119871 (120573lowast 120573)

120597120573lowast

1198961

120597120573lowast

1198962

= minus

119873

sum

119899=1

119869

sum

119895=1

119909lowast

1198951198961119899119875119895119899(119909

lowast

1198951198961119899minus

119869

sum

119894=1

119909lowast

1198941198962119899119875119894119899)

forall1198961 1198962 = 1 119870lowast

(6)

1205972119871 (120573lowast 120573)

12059712057311989511198961

120597120573lowast

1198962

= minus

119873

sum

119899=1

119909119895111989611198991198751198951119899(119909lowast

11989511198962119899minus

119869

sum

119894=1

119909lowast

1198941198962119899119875119894119899)

forall 1198951 = 1 119869 1198961 = 1 1198701198951

1198962 = 1 119870lowast

(7)

1205972119871 (120573lowast 120573)

12059712057311989511198961

12059712057311989521198962

=

119873

sum

119899=1

119909119895111989611198991199091198952119896211989911987511989511198991198751198952119899 if 1198951 = 1198952

minus

119873

sum

119899=1

11990911989511198961119899119909119895211989621198991198751198951119899 (1 minus 119875119895

2119899) if 1198951 = 1198952

forall1198951 = 1 119869 119896119894 = 1 119870119895119894

(8)


Let (120573lowast


119868 (120573lowast

120573) = 119872 sdot

1205972119871 (120573lowast

120573)

1205971205731205971205731015840 (9)



120573)]

minus1

= minus1

119872

[

[

1205972119871 (120573lowast

120573)

1205971205731205971205731015840]

]

minus1

(10)



119863119901-error = minus[

[

det1205972119871 (120573lowast

120573)

1205971205731205971205731015840]

]

minus1119870

(11)





3 Empirical Study



N

se1 (XI 120573)

seN (XI 120573)

0 10 20 30 40 50

Stan

dard

erro

r

(a) Larger samples

Stan

dard

erro

r

se1 (XI 120573)

seN (XI 120573)

seN (XII 120573)

N

0 10 20 30 40 50

(b) Better designs



119881car= 120573

car0+ 1205731TT

car+ 1205732TC

car

119881taxi

= 120573taxi0

+ 1205731TTtaxi

+ 1205732TCtaxi

119881bus

= 1205731TTbus

+ 1205732TCbus

(12)




(a)


120573taxi0

1205731 1205732


(b)
















Table 4 Continued


11 15 5 30 18 38 2 212 20 4 20 18 25 3 213 20 7 30 16 25 1 314 20 4 20 18 30 2 315 30 7 15 16 30 3 316 15 7 30 18 38 1 317 20 4 20 18 30 3 318 30 4 15 22 25 2 3



Orthogonal





119863-efficient





5 Conclusion






Acknowledgments


References



























Volume 2014




Journal of











Journal of


Function Spaces






Algebra










N

se1 (XI 120573)

seN (XI 120573)

0 10 20 30 40 50

Stan

dard

erro

r

(a) Larger samples

Stan

dard

erro

r

se1 (XI 120573)

seN (XI 120573)

seN (XII 120573)

N

0 10 20 30 40 50

(b) Better designs



119881car= 120573

car0+ 1205731TT

car+ 1205732TC

car

119881taxi

= 120573taxi0

+ 1205731TTtaxi

+ 1205732TCtaxi

119881bus

= 1205731TTbus

+ 1205732TCbus

(12)




(a)


120573taxi0

1205731 1205732


(b)
















Table 4 Continued


11 15 5 30 18 38 2 212 20 4 20 18 25 3 213 20 7 30 16 25 1 314 20 4 20 18 30 2 315 30 7 15 16 30 3 316 15 7 30 18 38 1 317 20 4 20 18 30 3 318 30 4 15 22 25 2 3



Orthogonal





119863-efficient





5 Conclusion






Acknowledgments


References



























Volume 2014




Journal of











Journal of


Function Spaces






Algebra
















Table 4 Continued


11 15 5 30 18 38 2 212 20 4 20 18 25 3 213 20 7 30 16 25 1 314 20 4 20 18 30 2 315 30 7 15 16 30 3 316 15 7 30 18 38 1 317 20 4 20 18 30 3 318 30 4 15 22 25 2 3



Orthogonal





119863-efficient





5 Conclusion






Acknowledgments


References



























Volume 2014




Journal of











Journal of


Function Spaces






Algebra










Acknowledgments


References



























Volume 2014




Journal of











Journal of


Function Spaces






Algebra
















Volume 2014




Journal of











Journal of


Function Spaces






Algebra









research article comparing the state-of-the-art efficient stated choice designs...

Documents