7 Experimentation and Test Markets

What is an Experiment?

• Experiment: research approach in which one variable (called an experimental, treatment, independent, or explanatory variable) is manipulated and the effect on another variable (referred to as a dependent variable) is observed.

• Example: the effect of advertising on sales or the effect of price on sales.– Advertising/price: experimental variable– Sales: dependent variable

Demonstrating Causation

• Causal research: research designed to determine whether a change in one variable likely caused an observed change in another.

• To demonstrate causation (that A likely caused B), one must be able to show 3 things:1. Correlation or concomitant variation2. Appropriate time order of occurrence3. Elimination of other possible causal factors

Demonstrating Causation

• Concomitant variation– To provide evidence that a change in A caused a

particular change in B, one must first show that there is correlation between A and B; in other words, A and B must vary together in some predictable fashion.

– This might be a positive (i.e. advertising and sales) or inverse relationship (i.e. price and sales).

Demonstrating Causation

• Appropriate time order of occurrence– To demonstrate that A caused B, one must be able

to show that A occurred before B occurred.

– Example: to demonstrate that a price change had an effect on sales, you must be able to show that the price change occurred before the change in sales was observed.

Demonstrating Causation

• Elimination of other possible causal factors– The most difficult thing to demonstrate in marketing

experiments is that the change in B was not caused by some other than A.

– Example: suppose a company increased its advertising expenditures and observed an increase in sales. It is possible that the observed change in sales is due to some factor other than increase in advertising: a major competitor might have decreased advertising expenditures, or increased price, or pulled out of the market, so one or a combination of other factors may have influenced sales.

Experimental Setting

Experiments can be conducted in a laboratory or in a field setting. Most experiments in the physical sciences are conducted in a laboratory setting; while many marketing experiments are field experiments.

• Laboratory experiments: experiments conducted in a controlled setting.

• Field experiments: tests conducted outside the laboratory in an actual environment, such as a marketplace.

Experiment Validity

• Validity: the degree to which an experiment actually measures what the researcher was trying to measure. The validity of a measure depends on the extent to which the measure is free from both systematic and random error.– Internal validity: extent to which competing explanations for

the experimental results observed can be ruled out.– External validity: extent to which causal relationships

measured in an experiment can be generalized to outside persons, settings, and times.

Field experiments offer a higher degree of external validity and a lower degree of internal validity than do laboratory experiments.

Extraneous Variables

Examples of extraneous factors/variables that pose a threat to experiment validity are:

• History: intervention, between the beginning and end of an experiment, of outside variables or events that might change the dependent variable.– Example: reading potential sales of Prego whie Ragu

is having its promotional campaign may not get the accurate sales reading.

• Maturation: changes in subjects occurring during the experiment that are not related to the experiment (a function of time) but that may affect subjects’ response to the treatment factor. It includes getting older, hungrier, more tired, and etc.

Extraneous VariablesExamples of extraneous factors/variables that pose a threat to experiment

validity are:

• Instrument variation: changes in measurement instruments (e.g. interviewers or observers) that might affect measurements.

• Selection bias: systematic differences between the test group and the control group due to a biased selection process.

• Mortality: loss of test units or subjects during the course of an experiment, which may result in a nonrepresentativeness.Example: in a study of music preferences of the population, if nearly all the subjects under the age of 25 were lost during the course of the experiment, then the researcher would likely get a biased picture of music preferences at the end of the experiment. As a result, the finding would lack external validity.

Extraneous VariablesExamples of extraneous factors/variables that pose a threat to experiment

validity are:

• Testing effect: effect that is a by-product of the research process itself.

Example: measuring attitude toward a product before exposing subjects to an ad may act as a treatment variable, influencing perception of the ad.

• Regression to the mean: tendency of subjects with extreme behavior to move toward the average for that behavior during the course of an experiment.

Example: the researcher might have chosen people for an experiment group because they were extremely heavy users of a particular product/service. In such situations, their tendency to move toward the average behavior may be interpreted as having been caused by the treatment variable when in fact it has nothing to do with the treatment variable.

Extraneous VariablesControlling Extraneous VariablesCausal factors that threaten validity must be controlled in some manner

to establish a clear picture of the effect of the manipulated variable on dependent variable.

4 basic approaches are used to control extraneous factors: • randomization: random assignments of subjects to treatment conditions to

ensure equal representation of subject characteristics.• Physical control: holding constant the value or level of extraneous variables

(i.e. age, income, lifestyle) throughout the course of an experiment.• Design control: use of the experimental design to control extraneous causal

factors.• Statistical control: adjusting for the effects of confounded variables

(extraneous causal factors) by statistically adjusting the value of the dependent variable for each treatment condition.

Experimental Design, Treatment, and Effects

• Experimental design: test in which the researcher has control over and manipulates one or more independent variables.

An experimental design includes 4 elements:1. The treatment, or experimental, variable (independent

variable) that is manipulated2. The subjects who participate in the experiment3. A dependent variable that is measured4. Some plan or procedure for dealing with extraneous causal

factors

Experimental Design, Treatment, and Effects

• Treatment variable: independent variable that is manipulated in an experiment.– Manipulation: a process in which the researcher sets the

levels of the independent variable to test a particular causal relationship.

– Example: to test the relationship between price (independent v.) and sales (dependent v.), a researcher might expose subjects to 3 different levels of price and record the level of purchases at each price level.

As a variable that is manipulated, price is the single treatment variable, with 3 treatment conditions/ levels.

Experimental Design, Treatment, and Effects

• Experimental effect: effect of the treatment variable on the dependent variable.

• Example: 3 different markets are selected to test 3 different prices (treatment conditions) for 3 months.Market Price Experimental effect (change in sales)

1 2% lower increased less than 1% 2 4% lower increased by 3% 3 6% lower increased by 5%

Limitations of Experimental Research

• High cost of experiments• Security issues

Competitors can decide whether and how to respond or stolen concepts that were being tested in the marketplace.

• Implementation problemsExample: a regional marketing manager might be very reluctant to permit her market area to be used as a test market for a higher price that might lower sales for the area.

Selected Experimental Designs• Pre-experiment designs

Designs that offer little or no control over extraneous factors. Studies using this designs are often difficult to interpret because they offer little or no control over the influence of extraneous factors.

– One-shot case study design: no pretest observations, no control group, and an after

measurement only (O1)X O1

– One-group pretest-posttest design: pre (O1)- and postmeasurements (O2) but no control group O1 X O2

X: the exposure of an individual/a group to an experimental treatmentO (observation): the process of taking measurements on the test units (individuals,

groups of individuals, or entities whose response to the experimental treatments is being tested: individual/group of consumers, retail stores, total markets, and etc.)

Selected Experimental Designs• True experimental designs

Research using an experimental group and a control group, to which test units are randomly assigned (R). These experimental designs are superior because randomization takes care of many extraneous variables.

– Before and after with control group design: involves random assignment of subjects or test units to experimental and control groups and pre – and postmeasurements of both groups. Experimental group: (R) O1 X O2Control group: (R) O3 O4

– After-only with control group design: involves random assignment of subjects or test units to experimental and control groups, but no premeasurement of the dependent variable. Experimental group: (R) X O1Control group: (R) O2

Selected Experimental Designs• Quasi-experiments

Studies in which the researcher lacks complete control over the scheduling of treatments or must assign respondents to treatments in a nonrandom manner.These designs frequently are used in marketing research studies because cost and field constraints often do not permit the researcher to exert direct control over the scheduling of treatments and the randomization of respondents.

– Interrupted time-series design: research in which repeated measurement of an effect “interrupts” previous data patterns. Ex: the use of consumer purchase panels to make periodic measurements of consumer purchase activity (Os) with introducing a new promotional campaign (X) .O1 O2 O3 O4 X O5 O6 O7 O8

– Multiple time-series design: interrupted time-series design with a control group, then researchers can be more certain in their interpretation of the treatment effect. Ex: an advertiser were testing a new ad campaign in a test city, that city would constitute the experiment group and another city (that was not expose to the new ad campaign) would be chosen as the control group.

Experimental group: O1 O2 O3 O4 O5 O6Control group: O1 O2 O3 O4 O5 O6

It is important the test and control cities be roughly equivalent in regard to characteristics related to the sale of the product (e.g. competitive brands available)

Test Markets

• Test market: real-world testing of a new product or some element of the marketing mix using an experimental or quasi-experimental design.

• Test-market studies are designed to provide info. regarding to the following issues:– Estimates of market share and volume.– The effects that the new product will have on sales of similar

products (if any) already marketed by the company. (cannibalization rate)

– Characteristics of consumers who buy the product.– The behavior of competitors during the test.

Test Markets

Types of test markets• Traditional (standard) test market: testing the product and

other elements of the marketing mix through regular channels of distribution which normally take 6 months or more.

• Scanner (electronic) test market: research firms have panels of consumers who carry scannable cards for use in buying particular products, esp. those sold through grocery stores. These panels permit us to analyze the characteristics of those consumers who buy and don’t buy the test products such as demographic data, pas purchase history, or even media viewing habits.

Test Markets

Types of test markets• Controlled test markets are managed by research suppliers who

ensure that the product is distributed through the agreed upon types and numbers of outlets. The research suppliers pay distributors to provide the required amount of shelf space for test products and carefully monitor sales of the product in these controlled test markets.

• Simulated test markets (STMs): simulations of the types of test markets noted above. They can be conducted more quickly than other approaches, at a lower cost, and can produce results that are highly predictive of what will actually happen.

Test Markets

Decision to conduct test marketing1. Weigh the cost and risk of failure against the probability of

success and associated profits2. Consider the likelihood and speed with which competitors

can copy your product and introduce it on a national basis.

3. Consider the investment required to produce the product for the test market vs. the investment required to produce the product in the quantities necessary for a national rollout.

4. Consider how much damage an unsuccessful new product launch would inflict on the firm’s reputation.

Test Markets

Steps in a test market study1. Define the objective of the test2. Select a basic approach (type of test market)3. Develop detailed test procedures (basic positioning

approach, the actual commercials to be used, pricing strategy, media plan, and etc.)

4. Select test markets5. Execute the plan6. Analyze the test results throughout the test period

(purchase data, awareness data, competitive response, source of sales)