RESEARCH METHODS

By Gordon A. Wyner

Experimental Design

Experimental design has been usedeffeetively in marketing research for manyyears to evaluate new produets andimproved versions of existing products.The goal is usually to find the best product,among the alternatives tested, based onsome criteria such as purchase, usage, orcustomer value.

Tn recent years, though, the applieatioushave evolved sotnewhat to reflect changingtnarketing issues. The need for increasedmarketing efficiency and effectiveness hasincrea,sed interest in experimentationbecause it has ibe potential to reduce cycletime for new products, rigorously evaluate awider range of potential product and serviceoptions to select tbe best one to pursue, andsystematically test many alternative com-petitive strategies. Experimentation canreduce the risk of costly ptx)duct failuresand increase the odds of targeting the rightcustomers with the right products.

Reseai'chers need to address several keyquestions about the current and future useof experimental design in marketingreseareh:

• What iU'e its distinguishing features?

• How is il besl applied in marketing?

Gordon A. Wyner isVice President ofMercer ManagementConsulting inLexington. Mass,

' How does it compare to othermethods?

• What are ils future prospects?

DiSTINGUtSHING FEATURES

Although experimental design sharessome features in common with other kindsof research, it also has some unique featuresthat differentiate it from other kinds of sci-entific investigations.

Testing things that don Y currentlyexist: One of the unique features is the cre-ation of a product, service, and/or eompeti-tive situation that isn't naturally occurringin tbe environment so it ean be evaluated.The need for experimentation is great whenhistorical information is not indicative offuture customers and products. This occurswhen firms enter new markets, such as along-distance lelephone cotnpany gettinginto tbe local service business or a bankgetting into online computer banking forconsumers. By definition, experimentationattacks a difficult problem to create some-thing that can't be observed without someintervention by the researcher.

Random assignment: Another uniquefeature is that the products to be tested arerandomly given to members of the eligiblepopulation. Tbe researcher controls whogets what product rather than allowing eus-tomers to self-seleet. This enables theresearcher to infer tbat the differences inresponse to different produets (ineludingtbe possibility ofa eontrol product) ai"ecaused by the products ratber than by selee-tion or some other variable.

Physical rather than statisticaleontrol: Tbe researeber goes to greatlengtbs t<i ensure that the products to betested meet explicit specifications that mustbe adhered to in administering the experi-ment to customers. In a tnte laboratory

experiment, physical control is at a maxi-mum. In marketing studies, whicb are usu-ally survey based or field tests, pbysicalcontrol is only approximated. Still llie goalis to rely heavily on physical controls andonly secondarily on statistical controls (i.e.,adjustitig for differences due to customervariability).

Independent variables: Like other typesof reseai'eh (e.g., regression analysis basedon bistorical data) some variables are con-sidered independent. The analytical objec-tive is to determine bow changes in tbesevariables affect some customer response. Inexperimental researcb, the key independentvariables—the different products to be test-ed—are actually manipulated by theresearcher.

Olher independent variables tbat areobserved as tbey naturally occur may beused as statistical controls, bul are generallyless important. They are sometimes usedsolely to reduce noise in the data so that theexperimental effeets can be estimated tnoreprecisely.

Experimental design plan: Unlike witbother types of research, experimental designgives the researeher tbe opportunity to cre-ate a plan that reflects all die different prod-ucts to be tested in ways that iire advanta-geous. Rather than accept the particularcombinations of the levels of the indepen-dent variables that already occur in the mar-ketplace (e.g., particulaj' product featuresand prices) the combinations that are mostinformative from a testing standpoint canbe built inlo the design for administration locustomers.

For example, in the curi'ent real environ-ment, the range of ihe independent vari-ables migbl be heavily restricted, while theexperimental design can test beyond thehistorical range. Often the independent vari-ables are so highly correlated in tbe currentenvironment that tbeir effeets cannot be iso-

MARKEriNG RESEARCH 39

lated. In the experimental design plan, theseindependent variables can be conslrueted tobe statistieally unrelated to eaeh other,enabling precise estimates of the sensitivityof response to each one.

The simplest plans involve two products,a test product and a control. More complexdesigns involve many products (e.g., 10new product concepts). Still more com-plex, and frequently encountered, is the fac-torial design, in which product attributes aredefined as factors and each attribute takeson multiple levels (e.g., the price for a PC isan attribute at four levels: $2,000, $2,100.$2,200, $2,300; memory is an attribute atthree levels: 8MB. 16MB, 32MB; andprocessor is an attribute at two levels: 133,166).

The factorial approach allows thereseareher to test a range of levels for con-tinuous attributes and infer wbat responsewould be at relevant levels in between (e.g.,a price of $2,050). It provides the chance toestimate the eombined eff"ect of severalattributes at once, sucb as price, processorspeed, internal memory, disk drive, andbrand name.

When the number of attributes and lev-els per attribute gets large, tbe total possiblenumber of combinations increases geomet-rically. Ratber than test all combinations, asubset can be selected under certain simpli-fying assumptions (e.g., no, or few, interac-tion effeets) while still allowing for statisti-cal estimation of the relevant parameters.This reduction of the number of experimen-tal combinations can be a powerful effieien-ey driver in the design process. Massivenumbers of producl combinations can beevaluated statistically from a relativelysmall subset of combinations that actuallygets tested using fractional factorialdesigns.

Another important design planning toolis blocking, wbich is a method for experi-mentally controlling error by segmentingresearcb subjects into homogeneous groups.It is also a vebicle for deciding which set ofproducts a specific customer will evaluatein cases where eacb person evaluates morethat one product (e.g., in choice modelingstudies). Blocking provides a systematicway to ensure that each person evaluates abalanced set of products (not all attractiveor unattractive).

Dependent variables: As with other

types of research, some variables must bespecified as the outcome variables tbat areaffected by the independent variables. Atthe outset of a test, tbe researcher stateswhich variables are expected to change as adirect result of the different produetsoffered, for example, stated interest in theproduct, purchase intentions, actual pur-chase, usage bebavior, image perceptions.

Population to be tested: All research isspecific to a particular universe of people orsegment of people in a market. The popula-tion must be speeifled because the effectsof the independent variables on tbe depen-dent variables may well be different foreaeb segment tested. The produet that isbest on average might not be the best onefor all individuals. Experimental design canbe used to find tbe best produet for particu-Uir types of customers, so it helps with tar-geting customers as well as tailoring theoffer.

Sample to be selected: A sample mustbe selected to achieve appropriate represen-tation of the relevant population and to pro-vide adequate precision to be able to detectthe effects of Ihe independent variables. Intbe simplest cases, sample-size determina-tion can be framed as testing for the differ-ence between the means of the test and con-trol products. The necessary sample sizeprovides the desired levels of type I andtype II error, and ean be derived from stan-dard statistieal formulas.

In most applied experiments, the samplesize issue is considerably more eomplexand less formulaic because typically thereare multiple outcome variables and oftenthe goal is to build a multivariate modelratber than simply to test hypotheses forstatistical significance.

Hypotheses to be tested: Good researchpractice requires a statement up front of themodel that links the independent to depen-dent variables. Usually the objective is notto test simple hypotheses, such as "Is prod-uct A superior to producl B?" Sometimesthe goal is to quantify the magnitude ofsuperiority of A over B, but usually it is todevelop a predictive model or simulationmodel to answer complex questions sucb as"Wbicb combination of product featureswill yield tbe best financial results?"

This calls for a functional form: Ts theresponse related to tbe independent vari-ables via a linear, quadratic, or logistic

equation? Are interactions between theindependent variables to be tested? Eorexample, does response to product-perfor-mance benefits depend on price?

If tbere are multiple, related outcomemeasures, how will the results be linkedtogether? For example, the experimentmigbt be used to determine tbe best productbased on expected customer profitability.This can be calculated from a composite oflikelibood of purchase, usage behavior overtime, and costs associated with tbe productand usage.

MARKETING APPEICATIONS

Experimental design has been usedextensively in product testing, trade-offanalysis, ajid market tests.

Product testing: Experimental design isfrequently used for consumer products,foods in particular. Eor example, for saladdressings, the amount of oil, salt, and sugarcan be systematically varied to identify theproduct that tastes best to consumers.

Eor some appiications, the hypothesis tobe tested is that there is a single ideal pointwhere all the attributes are at just the rightlevels. Any departure from this peak, forexample, introducing too much salt or notenough salt, leads to lower appeal.

A response surface model is appropriateto identify this peak from a range of levelsacross all attributes. Specific types ofexperimental designs (e.g., central compos-ite designs) are required for this purpose.Such designs enable tbe maximum to befound using a relatively small subset ofexperimental test product feature combina-tions.

Trade-off analysis: Experimentaldesign has been very effective for estimat-ing the impact of large numbers of productattributes (e.g., 10-20). The hypotheses tobe tested often inclttde competitive crosseffects (e.g., bow will eompetitor's pricecuts affeet our produet's demand?) and seg-ment differences (e.g., bow do businesscustomers value features compared to resi-dential customers?) as well as overallattribute effects (e.g., which attributes arekey drivers of demand?). Tbis leads to tbeneed for a decision support simulationmodel to combine these hypothesizedeffects and determine which product tooffer to whicb segment.

40 Fall 1997

The use of fi actional factorial and block-ing designs is critical to reducing the datacollection burden to a level that is accept-able to customers and cost-effective to exe-cute. For example, a design with 64 experi-mental combinations of attributes can beblocked into eigbt balanced sets of eightcombinations and assigned to samplegroups so tbat each person evaluates onlyone set of eight.

Market tests: Experimental design canachieve many of the same benefits that arerealized in producl tesling and trade-offanalysis in actual market situations. Thedistinctive quality of market tests—thatthey are conducted in the field under actualmarket conditions—usually constrains tbeflexibility of the design. For example, theeost of setting up a test market and tbe dif-ficulty of identifying comparable marketsreduces the number of test cells that arepractical to implement.

However, there are other instances inwhicb fewer practical constraints exists.Industries that rely beavily on direct mailand telemarketing cbannels can fairly easilyintroduce experimental design into tbeirregular eustomer contact process. This basbeen the case in the credit card and tele-phone services industries where productand service features, ad copy, and sellingscripts have been systemically varied to testhypotheses.

The power of experimental design canbe enhanced in this situation by linkingproduct design with customer targeting.Models for analyzing tbe data ean incorpo-rate product features and identifiable char-acteristics of individuals from customerdatabases. Tbe output then would ineludeprioritized lists of customers and the bestproduet alternative for each.

COMPARISON TO OTHER METHODS

To assess the value of experimentaldesign for partieular applieations, it is use-ful to evaluate it against otber methods onseveral criteria. The "competitive" altema-tives include:

• Descriptive sample surveys, in whicbcustomers are asked about tbeir atti-tudes, past behavior, and future inten-tions.

• Eield observation, in whicb customersare observed in their natural environ-ment without intervention fromresearchers wilh test offers or question-naires.

" Historical analysis of past behavior,which is typically captured in detailedpurchase or usage transaction records.

Some important evaluative criteria tbatdifferentiate between these methods are ran-domization, representativeness, and realism.

Randomization: This is the majorstrength of experimental design, whichenables strong causal inferenees to bemade. Historical analysis of behavior, fieldobservations, and surveys lack this featureand are, thus, on weaker ground in makingcausal inferences. Correlation, wbich canarise from many unidentified variables,does not prove causation.

Representativeness: Laboratory experi-ments rarely are eomprised of representa-tive samples because they often require thatsubjects be recruited from a small geo-graphic area and brougbt to a central loca-tion for data collection. Some form of con-venience sampling must be used, runningthe risk that the experimental sample differsfrom a truly random sample of the popula-tion. Would people in outlying areasrespond the same way if they had beenrecruited? Are those that come to centrallocations different from those that don't?

Despite this concern, experiments can bevery valuable because they estimate differ-ences in response between products so welleven if they estimate absolute levels ofresponse less accurately. Additionally, thereare situations in which customers can besampled on essentially a random basis fbrexperimental studies. Eor example, creditcard companies tbat select customers fordireet mail solicitation from tbeir own listsor from outside lists can be assured thattbose lists are accurately represented.

Realism: Experiments generally createan environment that is somewhat artificial;after all, the researcher must create it.Would customers respond the same way inan actual market situation compared to theexperiment? Would they exhibit tbe samelevels of price sensitivity, for example,wben tbey are spending real money out oftheir own pockets rather than responding to

hypothetical choice altematives?One way to deal with lack of realism is

to invest more resources into making tbeenvironment seem real to respondents. Useof simulated store sbelves (as in the originalpretest market forecasting approaches) andmultimedia environments to simulate morecomplex buying environments are ways ofadding realism.

It should be noted again tbat in marketswhere "controlled" channels such as tele-marketing and direct mail are used, theexperimental milieu may be virtually identi-cal to ibe real one. The eredit card customerwho receives several offers in the mailboxmay not be able to pereeive wbich ones areexperimental and wbich iire not.

FUTURE PROSPECTS

It is likely that experimentation will beused even more in the future than in thepast, for several reasons. A primary reasonis that one of its main benefits is efficiency,an increasingly important business objec-tive. It uses data parsimoniously to test spe-eifie bypotbeses in unambiguous ways,witb clear decision rules and action steps.Additionally, technology improvements canincrease the efficiency of conducting exper-iments and reduce some of the limitations,such as lack of representativeness and real-ism.

Several business trends favor experi-mental design over other methods. Tbedesire to measure results of marketing plansand activities fits well with experimenta-tion. The desire to create a continuous orga-nizational process of testing and learninglends itself to experimentation. And theneed to find new sources of businessgrowth bodes well for experimentationbecause it enables firms to test plans that gobeyond what they bave done in tbe past.

Finally, experimentation can be the basisfor answering "what if questions whicb arecritical for strategic and tactical decisions.Experimental design can measure whichamong a predefined set of altematives isbest 10 pursue. It represents a rigorous, sys-tematic approach to making tactical deci-sions compared to ad hoc, incrementalways. When tbe possible alternative sirate-gic scenarios can be anticipated, experimen-tation provides direct tests of tbe likely con-sequences of different actions. •

MARKETING RESEARCH 41