research design jeremy kees, ph.d.. stages in the research process determine research design analyze...

Research Design

Jeremy Kees, Ph.D.

Stages in theResearch Process

Determine Research Design

Analyze and Interpret the Data

Design Sample and Collect Data

Formulate Problem

Design Data Collection Method and Forms

Prepare the Research Report

Important Design Issues to Consider• How well do you understand the problem?

– Not very well = exploratory research needed

• What is your budget?– Low = exploratory research

• Do you have access to a large sample?– If not, exploratory research is the way to go

Important Design Issues to Consider• Do you need to understand how a specific

population feels about a particular issue?– Attitudes, evaluations, preferences, etc.?

• Are you trying to choose between options?– Advertising campaign ideas or specific ads

Overview of Research Design

Exploratory• “discovery”

Descriptive• “relationships”

Experimental• “cause-and-effect”

Typology of Marketing Research

• By Source– Primary

– Secondary

• By Methodology– Qualitative

– Quantitative

• By Objectives– Exploratory

– Descriptive

– Experimental (Causal)

Qualitative vs. Quantitative Research

• Depth versus Generalizability– Quantitative Methods

• Generalizations to other populations and/or situations

– Qualitative Methods• Rich Understanding

• Common Assumption:– Qualitative Data = preliminary– Quantitative Data = confirmatory

8

Qualitative Research

To gain a qualitative understanding of the underlying reasons and motivations

Small number of non-representative cases

Unstructured

Non-statistical

Develop an initial understanding

Objective

Sample

Data Collection

Data Analysis

Outcome

Quantitative Research

To quantify the data and generalize the results from the sample to the population of interest

Large number of representative cases

Structured

Statistical

Recommend a final course of action

Qualitative vs. Quantitative Research

Benefits of Qualitative Research

• Less expensive– Smaller sample size

• Opportunity to probe respondents– Observation of “real” customer reactions

– Great depth of information

• Can give you “direction” for decisions• Typically informs quantitative research

– Inexpensive/quick qual research can help tremendously in developing more extensive quant research

10

Descriptive Research

Exploratory Research

Causal Research

Relationship Among Research Designs

EXPLORATORY RESEARCH

12

• Focus groups: small group discussions led by a trained moderator

• Objectives:• Generate ideas• Understand consumer vocabulary• Reveal consumer needs, motives,

perceptions, and attitudes on products and services

• Understand findings from quantitative studies

Focus Groups

13

• In-Depth interview• A set of probing questions posed one-

on-one to a subject by a trained interviewer so as to gain an idea of what the subject thinks about something or why he or she behaves a certain way

In-Depth Interviews (IDIs)

Other Qualitative Methods

• Ethnographies– developing understandings of the everyday activities

of people in local settings

• Observation– Insight into actual, not reported, behaviors

• Mystery Shopping• Protocol analysis

– involves placing a person in a decision making situation and asking him or her to verbalize everything he or she considers when making a decision

“Alternative” Techniques

• Implicit Association Test– https://implicit.harvard.edu/implicit/

• Eye Tracking

• Resistometer

DESCRIPTIVE RESEARCH

Descriptive Research

• For our purposes, “survey” research– Asking a sample of people from a population a

set of questions

– Using the answers to describe that population

• Common Goals– Describe what is going on or exists

– Estimate how groups of consumers might behave

– Examine relationships between two or more variables

– Predict

0.00 500.00 1000.00 1500.00 2000.00 2500.00 3000.00

calactrc

-2000.00

-1000.00

0.00

1000.00

2000.00

ca

ldif

f

Actual Calories Consumed

Differences between Estimated and Actual Calories

Descriptive Research

• In contrast to exploratory (qualitative) techniques, the purpose of survey research is to produce statistics

• In contrast to exploratory (qualitative) techniques, survey research is concerned with generalizability– Sampling becomes very important

Descriptive Research

• Three key aspects– Designing questions– Sampling– Data collection

• We want to have a study that is precise, credible, and accurate– A key issue is reducing error throughout the

process (Fowler)

• Poor attention to ANY of these aspects can result in poor results (Fowler, Ch. 13)

Descriptive Research

• Two Basic Types1. Longitudinal

1. FUF

2. CWL

2. Cross-Sectional1. NFI

FOP Segmentation

Measures for Nutrition Knowledge and Nutrition Motivation are defined

Nutrition Knowledge

Nutrition Motivation

Moms are assigned into nine groups based on their level of Measures for Nutrition Knowledge and Motivation

The nine groups are collapsed into six segments to allow for effective marketing and communications

Index scores are constructed to place the six segments in the involvement tree

Compenent 1

Compenent 2

Compenent 3

Compenent 4

Compenent 5

Compenent 1

Compenent 2

Compenent 3

S1 S2

S3 S4

S6

S5

FOP Segmentation

SEGMENT 2 DESCRIPTION

• 15% of mom population

• More kids = large HH

• Older kids (ages 11-15)

• Diverse (mostly Hispanic)

• Many Spanish speakers

• Nutrition content is imp in food purchase decisions

• Urban

• South

• Low education (completed some HS)

• Likes to shop at Whole Foods

• Lower income (<$25K)

Low KnowledgeModerate to High Motivation

Kn

owle

dge

able

Motivated

Low

Hig

h

Low High

FOP Segmentation

High KnowledgeHigh Motivation

Kn

owle

dge

able

Motivated

Low

Hig

h

Low High

SEGMENT 6 DESCRIPTION

• 8% of mom population

• Mostly white

• Feel knowledgeable about nutrition

• Exercise the most of the segments

• West

• Highly educated (college+)

• Shops at Trader Joe’s and Whole Foods

• Use Nutrition Facts Panel more than other segments

• Higher income ($100K+)

Limitations…

• Can we infer causation from a strong correlation?

• Survey research is limited by what people are willing and able to tell us in the context of a survey

• This limitation can be addressed using other methods (triangulation)

EXPERIMENTAL RESEARCH

Experimental Research

• Helps us determine if one or more IVs (treatment, predictors) causes or affects one or more DVs (outcome variables)

• Most demanding design—strongest conclusion

• Requires the highest degree of understanding of the problem

Example….

• Do Cigarette Warning Labels work?– CBS

CWL Experiment (controls)

CWL Experiment (treatments)

Trade-Offs in Types of Experimental Designs

NFI Project

•Between-Subjects– Strongest design to get “true” effects

– Toughest to get significant differences across treatment conditions

•Within-Subjects– Prone to demand effects

– Most likely to get significant differences across treatments

•Client satisfaction vs. rigor

Evidence that supports a causal inference…• Correlation

– Observe the relationship (variability) between ad spend and sales

• Independent variable (treatment) occurs before the dependent (outcome) variable– Change ad spend and look for an effect on sales.

• Eliminate “Alternative Explanations”– If we observe an increase in sales when we increase

ad spend….• Were there changes in any other parts of our marketing mix?

• What is going on with the economy?

• Any changes in competition’s ad spend?

• Etc, etc, etc……

Causal Research (Experimental Design)

• Internal Validity

Causal Research (Experimental Design)

• External Validity

To conclude…

• Experiments are the only way to show causation– But often take a back seat to descriptive

studies due to time, cost, and control issues

• Exploratory and descriptive studies are useful, but be careful not to infer too much– Correlation is not causation

• Again, let your research questions dictate your design!

Some practical issues….

• Qualtrics Research Platform– Free you under VSB’s “site license”– Extremely user friendly, but also very

robust– www.qualtrics.com

Some practical issues….

• Amazon Mechanical Turk– The most inexpensive way to collect

consumer data– Extremely user friendly, but also very

robust– www.mturk.com

Online Survey (created by you and housed on Qualtrics’ server)

Create HIT (Human Intelligence Task) on Mturk•Description of your study and a (Qualtrics) link to it

Mturk workers (survey responders) “work” on your HIT (i.e., they take your survey)

Data is recorded by Qualtrics. Participants who complete the survey are given a code to input into Mturk. Those that enter a valid code, get paid.

Everyone is happy

Team Assignment #2

• Refine your research questions– Need to be clear, concise, and “testable”

• Based on your research questions1. Design 2 potential studies that could

address your research questions1. Explain the benefits and weaknesses of each

approach

2. Pick the “best” design and explain your decision

(Note: Don’t worry about measurement or sampling too much---you’ll have your chance to do that later)