making words speak: leveraging consumer insights from ... · making words speak: leveraging...
TRANSCRIPT
Making Words Speak: Leveraging Consumer Insights from Online Review
Text to Improve Service Quality
Andrea Ordanini Bocconi University
Raji Srinivasan University of Texas at Austin
Anastasia NanniBocconi University
MSI Managerial Presentation
The Research Idea
2
How do managers make sense of online reviews?– Ratings are important but they cannot be highly diagnostic
(e.g., j-shaped distribution)
– Texts are potentially more informative but they are unstructured and thus challenging to exploit
Managers might benefit from textual information in structured form
Textual information in structured form should: – increase the managerial ability to change and improve the
offering,– especially when the manager has proper motivation to act,
and the organizational context provides adequate opportunity
MSI Managerial Presentation
Textual Data in Structured Form
3
Providing structure to textual data means:– Visual representation
• Thematic Maps
– Redundancy• Different categorization
– Dual Coding • Pictorial and Tabular Forms
Research Question - Does treating managers with textual data in structured form help to improve their results?
MSI Managerial Presentation
Research Approach
4
To avoid self-selection bias, unobserved heterogeneity issues, and given the time lag to detect the potential effect, we investigate our RQ through a Randomized Control Trial
Despite its challenges, RCT has been recently advocated in Marketing to increase the confidence in some strategic choices
MSI Managerial Presentation
Method
5
Jan 16 Mar 1st 16 Jun 1st 16 Aug 30th 16
Step 1Sampling
Step 2Start Treatment
Step 3Reinforcement
Step 4End Treatment
TREATMENT PERIOD
Perf. pre06/08 2015
Perf. post06/08 2016
Step 5Post- Hoc
Nov 16
n = 598 n = 203 n = 146 n = 135 n = 99
Primary Data
Stimulus andSecondary Data Stimulus Primary and
Secondary DataPrimary
Data
We focus on Italian hotel industry and the Tripadvisor context (for relevance)
We ensure collaboration from Federalberghi (for sampling) and a leading reputation mgmt company (for measures)
MSI Managerial Presentation
Step 1 Sampling
6
Hotel characteristics Participant Non-Participant Difference
Segment c2(2) =.42; p=.81
Budget 19% 21% Mid-price/Suite 57% 57%
Boutique 24% 22% Location c2
(1) =.38; p=.54 Urban 50% 47%
Touristic 50% 53% Size c2
(1) =.05; p=.82 <30 rooms 52% 51% >30 rooms 48% 49%
Respondent characteristics Participant Non-Participant Difference
Role c2(2) =2.56; p=.28
Owner 60% 64% Hotel Manager 32% 31%
Marketing Manager 8% 5%
Education c2(3) =3.87; p=.28
Primary 35% 29% Secondary (Hotel) 43% 44%
Secondary (General) 19% 20% Tertiary 3% 6%
Age 50.0 48.1 t=1.91; p=.06 Work experience (yrs) 3.2 3.5 t=1.80; p=.07
Perceptions on reviews (1-7) Participant Non-Participant Difference
Reviews Credibility 3.9 3.7 t=1.64; p=.10 Reviews are Informative (score) 4.0 3.8 t=1.60; p=.11 Reviews are Informative (text) 3.9 4.1 t=1.38; p=.17 Decision Confidence 4.1 4.0 t=0.39; p=.70 Reviews Usefulness 4.7 4.4 t=1.99; p=.05 Lack of Time for Reviews 3.3 3.6 t=1.49; p=.14 Lack of Staff for Reviews 3.5 3.6 t=0.90; p=.37
MSI Managerial Presentation
203 hotel managers participated to our study:
- 101 in the control condition
(report with text of the last year’s reviews
of their hotels and the ratings)
- 102 in the treatment condition
(wordcloud and sentiment analysis table from
last year’s reviews of their hotels, in addition to
what provided to managers in the control condition)
Steps 2: Stratified Randomization
7
Treatment
and
Typology
Location and Size
Urban Locations Touristic Locations*
Less than 30 rooms
More than 30 rooms Total
Less than 30 rooms
More than 30 rooms Total Total
Control
1-2 Stars 10 2 12 7 - 7
3 Stars 15 21 36 17 7 24
4-5 Stars 1 8 9 1 13 14
Total 26 31 57 25 20 45 102
Treatment
1-2 Stars 10 3 13 4 - 4
3 Stars 15 19 34 19 7 26
4-5 Stars 1 9 10 2 12 14
Total 26 31 57 25 19 44 101
MSI Managerial Presentation
Steps 2/3 – Example of Treatment (wordcloud + sentiment table)
8MSI Managerial Presentation
Steps 4 – Analysis
9
Our design reflects a Split-Plot model, where the effect of interest – the treatment effect - is the coefficient “Treatment x Year”
MSI Managerial Presentation
Online Ratings Coef. SE* P>|t|
Treatment (vs. Control) -.102 .114 0.37
Year (2016 vs. 2015) .007 .060 0.91 Treatment#Year
Treatment x Year .218 .095 0.03
Constant 4.186 .078 0.00
Random-effects
Parameters Estimate SE* [95% Conf. Interval]
Hotel: Id
Var (_cons) .294 .052 .207 .416
Var (Residual) .084 .014 .059 .117
Steps 4 – Analysis: Main Result
10
+5.4%
MSI Managerial Presentation
The managerial use of
online review text
analytics improves
service quality by
5.4% during the
treatment window,
comparing hotels in
the treatment vs.
control conditions
4.1
4.15
4.2
4.25
4.3
Aver
age
Perfo
rman
ce -
Rev
iew
s Sc
ores
2015 2016Year
Control Treatment
Average Treatment Effect
+5.4%
Heterogeneous Treatment Effect: Perceived accountability (Opportunity)
11MSI Managerial Presentation
The positive effect
of review text
analytics
disappears when
managers do not
feel accountant-.10
.1.2
.3P
erfo
rman
ce -
Var.
(Pos
t - P
re)
4 5 6 7Perceived Accountability Pre-Treatment
Control Treatment
Average Treatment Effect * Perceived Accountability
22%ß à 78%
Heterogeneous Treatment Effect: Perceived Performance ‘Pre’ (Motivation)
12MSI Managerial Presentation
The positive effect
of review text
analytics also
disappears when
managers are
satisfied with the
existing level of
performance
-.20
.2.4
.6Pe
rform
ance
- Va
r (Po
st-P
re)
1 2 3 4 5Perceived Performance Pre-Treatment
Control Treatment
Average Treatment Effect * Perceived Performance
69%ß à 31%
Main Implications
13
Textual data in structured form seems to provide a positive effect on online performances:– It increases the quality of the offering– It generates positive customers responses
The effect size is not trivial and is robust to several alternative assumptions
Decisional stimuli and organizational characteristics however act as important boundary conditions for such positive effect
The main benefit for managers from the use of textual data in structured form is to clearly and rapidly identify points of weakness in their offering
MSI Managerial Presentation