adapting conjoint to the mobile phenomenon
TRANSCRIPT
Chicago New York London Dubai New Delhi Bangalore SingaporeSan Francisco
Making Conjoint MobileAdapting Conjoint To The Mobile Phenomenon
Chris Diener, Rajat Narang, Mohit Shant, Mukul Goyal, Hem Chander
Sawtooth Conference 2013
2
Evolution of Market Research
Paper and Pen
Personal Computer
Laptop Computer
Smartphones/Tablet
Change in platforms followed increased mobility
Over the years, Market Research has adapted to various survey platforms
3
Smartphones sales went up by 33% in 2012.
Sale of Internet enabled phones in emerging markets will reach 1.2 billion by 2015, which will add 500M new internet users.
Both India and Africa average three mobile phones for every four people.
Number of tablet users are growing every year, adding to more mobility to lifestyle.
Why Research is going Mobile
Smartphone and mobile web penetration is growing day by day
4
Dawn of the era of Mobile Research
*“Catch the Wave” paper by Steve von Bevern, Research Now
“65% of the researchers expect to allocate up to 10% of their budgets to mobile”
Respondents prefer Smartphone surveys, primarily due to its “on the go” nature
25%18%
31%
4%
18%
0%
10%
20%
30%
40%
UK US Australia Canada Total
“I have run multiple research projects via mobile”
Market Research companies are adapting to this phenomenon the world over
5
App Based SMS Based
Web Browser Based
Different ways of conducting surveys on mobile platforms
6
Customer Satisfaction Studies
Ad Value Identification
Media & Message Effectiveness
Habits and Usage Studies
Customer Service Representative (CSR)
Concept Testing
Customer Attitude and Expectation
“For some projects with extensive or complex stimuli, e.g., many discrete choice studies, survey completion on smartphone might be ill advised”- Mobile Research Risk, Burke Inc, 2013
BUT!There is discernible skepticism for conducting conjoint on mobile platform
Different types of surveys using mobile platforms
Researchers are utilizing mobile platforms to conduct different types of short surveys
7
Complexity Representation on small screen
Short attention span
Too many tasks Large number of attributes in a survey
Smart phone penetration difference
Challenges in conducting conjoint on mobile platform
8
Adapting Techniques
– Back to Basics: Tested Pairwise Ratings and Partial Profile, which are simpler to evaluate
– Less is more: Fewer attributes per screen
– Tailor Made Methodologies: New techniques tailor-made for mobile platform to increase interactivity and ease of understanding like Shortened ACBC
Reduction ofattributes
Visual Transformations
– Simplify
– Simplification of tasks, leading to optimal use of screen space
– Reduction of concepts on screen
– Minimum scrolling to view tasks
– Optimize
– Coding optimized to fit the tasks on mobile screens
Reduction of concepts on screen
Possible Resolutions
9
US India– Topic: Determine the preferred multimedia tablet configuration
– Number of attributes tested: 9
– 8 random and 2 fixed tasks per respondent
– Platform - Desktop / Laptop (referred here as PC)
- Mobile / Tablet (referred here as Mobile)
– Census representation of Sample MethodologyPlatform
PC Mobile
CBC (3 concepts per screen) 200 200
PCR (Similar to CVA) NA 200
CBC (2 concepts per screen) NA 200
Partial Profile NA 200
Shortened ACBC NA 200
*We thank IndiaSpeaks and uSamp for providing the sample in India and US respectively
Conducted a Multi Country Survey
10
Heading 24pt Bold Calibri
PCR – Pairwise comparison rating
CBC Mobile (2 concepts)
CBC Mobile(3 concepts)
Partial Profile Shortened ACBC
–Similar to CVA technique in data collection
–Estimation done in two ways– Discrete Estimation– Chip Allocation
–2 concepts per screen (excluding none)
–3 concepts per screen (excluding none)
–Similar to CBC on PC
–6 attributes evaluated perconcept
–2 concepts per screen
–Only Pre-screening and consideration set sections
–Can we go back to the basics?
– Is less better?
–What is the impact of Visual Transformation?
–Do lower number of attributes improve respondent’s ability to focus?
–How well can simpler tasks predict respondents choice?
Rat
ion
ale
to t
est
De
scri
pti
on
Evaluated the following techniques on Mobile and compared them with CBC on a PC
11
‒ 9 point rating‒ Rating
responses transformed to:‒ Discrete
Choice‒ Chip
allocationfor estimation
‒ No frills, clear display of concepts
‒ Acts as None
‒ 2 Concepts per screen
PCR – Pairwise comparison rating
12
‒ None option shown at the bottom
‒ More width per concept
‒ Higher Font Size
CBC Mobile (2 concepts)
13
‒ Optimized Coding to fit on one screen
‒ Optimal use of real estate
‒ None option shown at the bottom
CBC Mobile (3 concepts)
14
‒ Only 6 attributes per screen
‒ Brand and price on every screen
‒ None option shown at the bottom
‒ Clear display of concepts with high font size
Partial Profile
15
Pre
scre
en
ing
Ne
ar N
eig
hb
orh
oo
d C
on
cep
ts
Scre
en
ing
Task
s
Bu
ild Y
ou
r O
wn
‒ Prescreening of attributes
‒ Simplified BYO task
‒ Well defined concepts
‒ 7 attributes in a concept
Shortened ACBC
16
– Robustness of results on mobile as compared to those on PC
– Adaptation of various techniques to the mobile platform
Researcher Perspective
– Average Time Taken
– Readability
– Ease of understanding
– Enjoyability
– Encouragement to give honest opinions
– Realism of tablet configuration
Respondent Perspective
Evaluation of research from multiple viewpoints
17
Results – Researcher’s Perspective
18
Technique
CBC PC 1 1
PCR – Chip Allocation 0.44 0.57
PCR – Discrete Estimation 0.64 0.79
CBC Mobile (2 concepts) 0.77 0.75
CBC Mobile (3 concepts) 0.84 0.91
Partial Profile 0.83 0.78
ACBC 0.79 0.71
Correlation Analysis with utilities of CBC PC
Correlation analysis shows most of the methods are similar to results of CBC PC
19
Technique
CBC PC 63.5% 58.1%
PCR – Chip Allocation 53.9% 56.5%
PCR – Discrete Estimation 64.5% 66.1%
CBC Mobile (2 concepts) 82.2% 58.0%
CBC Mobile (3 concepts) 64.8% 54.6%
Partial Profile 78.5% 67.4%
ACBC NA 79.2%
Hold Out Task Accuracy
Statistical Significant Difference from CBC PC at 95% confidence level for each country
– Hold out tasks placed in middle and end of exercise
– They were of similar format as conjoint exercise
Hold Out task prediction rates are all in “well accepted” range.
4 Methods do better in hold out task when compared with that of PC
20
Technique
CBC PC 0.09 0.10
PCR – Chip Allocation 0.27 0.29
PCR – Discrete Estimation 0.06 0.06
CBC Mobile (2 concepts) 0.04 0.10
CBC Mobile (3 concepts) 0.09 0.11
Partial Profile 0.05 0.09
ACBC 0.10 0.10
MAE Analysis
MAE tells a similar story with 3 methods on mobile doing better than CBC on a PC
21
Results – Respondent’s Perspective
22
3 3.33.7 3.7 3.8
4.5
3.13.6
4.9
3.74.2
5.8
0
1
2
3
4
5
6
7
CBC PC CBC Mobile (2 Concepts)
PCR CBC Mobile (3 Concepts)
Partial Profile ACBC
In M
inu
tes
US
India
Avg. Time taken to complete trade-off exercise
Respondents take more time on mobile devices, but simpler tasks are quicker, for example CBC on Mobile – 2 tasks
ACBC, takes slightly more time, even more so in India, indicating that Indian market is not that familiar with these techniques
23
82% 85% 81% 76%85%
73%64%
52% 53%
72%
56% 52%
0%
20%
40%
60%
80%
100%
CBC PC PCR CBC Mobile (2 concepts)
CBC Mobile (3 concepts)
Partial Profile ACBC
US
India
Readability
Statistical Significant Difference from CBC PC at 95% confidence level for each country
Readability is not an issue and respondents find the survey legible on mobile
24
Ease of Understanding
81% 79%73%
62%69% 69%73% 78% 74%
68%76% 72%
0%
20%
40%
60%
80%
100%
CBC PC PCR CBC Mobile (2 concepts)
CBC Mobile (3 concepts)
Partial Profile ACBC
US
India
Statistical Significant Difference from CBC PC at 95% confidence level for each country
Ease of understanding is comparable to that of PC for simpler tasks across both countries. India scores well on ease of
understanding for all techniques
25
Enjoyability
73%
59% 55%48%
58% 53%
68%78% 75%
65%74%
80%
0%
20%
40%
60%
80%
100%
CBC PC PCR CBC Mobile (2 concepts)
CBC Mobile (3 concepts)
Partial Profile ACBC
US
India
Statistical Significant Difference from CBC PC at 95% confidence level for each country
Indian respondents enjoyed the survey on mobile more than their US counterparts
26
76% 75%64% 61%
71% 73%73%81% 82%
75%87%
81%
0%
20%
40%
60%
80%
100%
CBC PC PCR CBC Mobile (2 concepts)
CBC Mobile (3 concepts)
Partial Profile ACBC
US
India
Encouragement to give honest opinions
Statistical Significant Difference from CBC PC at 95% confidence level for each country
Majority felt that they were encouraged to give honest opinions for all the methods evaluated on mobile platforms
27
75% 75% 73%63%
71% 67%69%77% 75% 72% 72% 74%
0%
20%
40%
60%
80%
100%
CBC PC PCR CBC Mobile (2 concepts)
CBC Mobile (3 concepts)
Partial Profile ACBC
US
India
Tablet configuration looked realistic
Statistical Significant Difference from CBC PC at 95% confidence level for each country
Tablet configuration, which was similar for all techniques, gets very similar response from all respondents
28
Implications
29
Yes it is POSSIBLE!
Conjoint surveys on mobile platform provided robust data quality
Respondents could read and answer comfortably on mobile platforms
Respondents took slightly longer to complete the surveys on mobile platforms
Respondents felt that the product combinations looked realistic to them on mobile platforms
30
By combining simple techniques with basic aesthetics!
All methods customized for mobile platform performed well. Simplicity and optimal use of screen gives great results
“3 Tasks concept” should be avoided on mobile platform as respondent perceptions are less favorable -- though accuracy parameters do not differ much
“2 Tasks concept” is the preferred layout for mobile platform. Both full profile and partial profile perform well. However, full profile is easier to understand
PCR, when done through discrete estimation, performs well. ACBC, even though it gives good accuracy, perhaps should be avoided as it takes longer and scores lower on “Ease of understanding"
31
3232
If you need help with Analytics or Research, please write to us:[email protected]@[email protected]
For Media related queries [email protected]
For all other queries [email protected]
32
HEAD OFFICE
314 Marble Arch Tower,55 Bryanston Street,
London W1H 7AA
Phone : + 44 207 868 2240
UK OFFICE
DLF Cyber City SEZ,Building#14, 4th Floor, Tower B,DLF Phase-III, Sector 24 & 25A,
Gurgaon-122002,
Phone: +91.124.4953.400
INDIA OFFICE
AbsolutData AnalyticsMiddle East JLT
Office 1604, Tower BB1Mazaya Business Avenue
Jumeirah Lake Towers
Phone: +97150-1577257
DUBAI OFFICE
1851 Harbor Bay Parkway,Suite 125, Alameda,
California, USA – 94502
Phone: +1 510 748 9922Fax: +1 510 217 2387