affect reflection technology in face-to-face service

Affect Reflection Technology in Face-to-Face Service Encounters

by

Kyunghee Kim

B.S., Korea Advanced Institute of Science and Technology (2007)

Submitted to the Program in Media Arts and Sciences,School of Architecture and Planning,

in partial fulfillment of the requirements for the degree of

Master of Media Arts and Sciences

at the

MASSACHUSETTS INSTITUTE OF TECHNOLOGY

September, 2009

© Massachusetts Institute of Technology 2009. All Rights Reserved.

ARCHIVES

MASSACHUSETTS INSTITiUTEiOF TECHNOLOGY

OCT 2 6 2009

LIBRARIES

Signature of Author: ............................... .............. Kyunghee Kim/ y Kyunghee Kim

Program in Media Arts and SciencesAugust 7, 2009

Certified by:........... ..................... . ....................................Rosalind W. Picard

Professor of Media Arts and SciencesProgram in Media Arts and Sciences

Accepted by: .................. ..... ....... ... ... / - .Deb Roy

Chairman, Departmental Committee on Gaduate StudiesProgram in Media Arts and Sciences


by

Kyunghee Kim

Submitted to the Program in Media Arts and Sciences,School of Architecture and Planning,

on August 7, 2009 in partial fulfillment of therequirements for the degree of

Master of Science in Media Arts and Sciences

Abstract

This thesis examines the role of facial expressions in dyadic interactions between a bankingservice provider and customer. We conduct experiments in which service providers manipulate theirfacial expressions while interacting with customers in one of three conditions: In the neutralcondition the banker tried to maintain a neutral facial expression; in the smiling condition the bankertried to smile throughout the interaction; in the empathetic condition the banker tried to respond withthe same or complementary facial expressions. Results show that the customers (n=46) were moresatisfied with the interaction when they perceived the service provider was empathetic. Moresignificantly, the service provider and customer shared synchronized facial expressions with manyprolonged smiles, when customers said the service provider was empathetic. We suggested threedifferent criteria to investigate customer satisfaction as follows; according to what the serviceprovider tried to convey, what the customer perceived and what was actually detected in theirinteractions. According to the analysis of the interactions, smiling bankers who shared smiles wereevaluated as the best while smiling bankers who did not share smiles with customers were appraisedsimilar to non-smiling bankers.

Thesis Supervisor: Rosalind W. PicardTitle: Professor of Media Arts and Sciences


by

Kyunghee Kim

Thesis Reader: ..................... .............................................. ...Deb Roy

Associate Professor of Media Arts and SciencesMIT Program in Media Arts and Sciences


by

Kyunghee Kim

Thesis Reader:......Dan Ariely

James B. Duke Professor of Behavior EconomicsDuke University


by

Kyunghee Kim

Thesis Reader: ....................David Price

Executive-in-Residence, Bank of AmericaMIT Media Lab

Acknowledgements

I would like to thank my advisor, Rosalind W. Picard, who supported me running interesting

psychology experiments with her constant interest and feedback over a long period and encouraged

me not to give up anything. I also would like to thank my colleagues in the Affective Computing

group; Jackie Lee who always listened to my words and gave me good advice; Ming-Zher Poh who

patiently and cheerfully waited for me for our heart phones project; Micah Eckhardt who showed me

MIT is not only the place for technology but also the place full of cool potters; Rob Morris who did

not hesitate to volunteer his girlfriend to help me label a lot of DVDs; and I appreciate my other lab

mates for having been good colleagues: Hyungil Ahn, Shaundra Daily, Elliot Hedman, Ehsan Hoque,

Rana el Kaliouby.

I could not thank my friend, Nandi Bugg, who was working with me as an undergraduate

researcher, more or less than others for her hard work to run the experiments together and clean up of

the data. Without her exact time keeping habits, this thesis would not have been able to be born. I

thank and miss Robin Glancy who spent lots of his time off-work interacting with customers. I hope

to see his father's masterpieces at the Museum of Fine Arts some day. I also enjoyed working with

Adnai Mendez who amused me with his funny jokes and stories about his first baby. I thank

Stephanie Carpenter who helped me to reserve the lab space and recruit an ethnically diverse group

of people so that the experiment did not have just my Korean friends. I bet she will do a great job in

the Ph.D management department at the University of Michigan.

I love my Korean friends who entertained me with their hilarious lives and bugged me to

have boba tea at Allston while I was exhausted. Especially Sungmin Kim, you have been the best

roommate ever.

I have had a wonderful summer in Cambridge because of all of you. And to my family; Even

if we are away I always listen to your love and I love you the most.

To my family

Contents

1. Introduction.................................................................20

1.1 B ackground ................. . ................ ........................................... .................. 21

1.2 Overview ......................... ..................... 23

2. Experiment Design................................................ ............. ................... 24

2.1 General Description of the Experiment .................................................................. 24

2.2 Participants - Hired Bankers .............................................. ...... ..... 26

2.3 Participants - Customers .............................................................. 27

2.4 Procedures .................... ................................. 27

2.5 Facial Expressions Coding .................. ................................................... . .......28

3. M easures ........................................................................................................ 31

3.1 Facial Expression Measures ................... . ...................... 31

3.2 Customer Satisfaction Measures .................. ....................................................... 32

4. Banker M anipulation...........................................................34

4.1 Manipulation Checks ..................................................... ........... .. ............ 34

4.2 Banker and Customer Perspective Differences ......................................................... 44

5. First Perspective Labeling and Second Perspective Labeling......................................46

5.1 Data Format ............................... .... ... . . ............... ....... 47

5.2 Agreement between First and Second Perspective Labeling ........................................ 50

6. Customer Satisfaction Data Analyses................................................................... 55

6.1 Hypotheses............................. ...................................... 55

6.2 Analysis 1. Cognitively Reported Customer Perception .............................................. 56

6.3 Analysis 2. Banker Manipulation ........................... .... .......... 57

6.4 Analysis 3. Affectively Measured Synchronized Facial Expressions ................................. 58

6.5 Summary .................... ..... ..................... 66

7. Dyadic Communication Data Analyses................................................................68

7.1 Hypotheses................. .................................... ............ 68

7.2 Analysis 1. Cognitively Reported Customer Perception ........................... 69

7.3 Analysis 2. Banker M anipulation .................................... ... ...................... 73

7.4 Analysis 3. Affectively Measured Synchronized Facial Expressions ................................. 76

7.5 Summary ........................................................................................... 79

8. Specific Classification and Analysis of Smiles ........................................ ..... 80

8.1 Smiles in Context ......................................................................... 80

8.2 Validation of Social, Genuine and Greeting Smile .................. ................................. 81

9. Conclusions and Future Work ............................................................................. 83

9.1 Summary ................................. ................... ............... .......... 83

9.2 Suggested Future Work .............................. .......................................... 84

Appendix A: 361 Banker and Customer Facial Expressions Synchrony IDs ........................86

Bibliography................................................. ......................................... 87

List of Figures

Figure 2-1. nonreciprocal one-with-many design ........................................................... 25

Figure 2-2. Setting where banker and customer interact .................................................. 26

Figure 2-3. Labeling interface used by banker and customer ............................................. 29

Figure 2-4. Labeling interface used by four outside coders ............................. 30

Figure 4-1. Banker Manipulation Checks ...................................... ....................... 35

Figure 4-2. Histogram of Measurement 1.The percent of smile .......................................... 39

Figure 4-3. Histogram of Measurement 2.The percent of neutral facial expressions ................... 40

Figure 4-4. Histogram of Measurement 3.The number of banker's transitions ........................... 41

Figure 4-5. Histogram of Measurement 4.The difference between banker's transitions and customer's

transitions ......................................................................................... 42

Figure 4-6. The number of smile and The percent of smile ..................... ..................... 43

Figure 4-7. The number of neutral facial expressions and the percent of neutral facial

expressions ..................................... ............................... 43

Figure 5-1. Banker's Labeling of Banker's DVD .................................. 48

Figure 5-2. Customer's Labeling of Banker's DVD ..................................................... 48

Figure 5-3. Customer's Labeling of Customer's DVD .................................. .................... 49

Figure 5-4. Banker's Labeling of Customer's DVD ..................... .................... 49

Figure 5-5. Banker's Labeling of Banker's DVD and Customer's Labeling of Banker's DVD.......50

Figure 6-1. Interaction satisfaction, Information satisfaction and Empathy score ...................... 56

Figure 6-2. Interaction satisfaction, Information satisfaction, Empathy score ............................ 57

Figure 6-3. Classification Rules ............................................................ 59

Figure 6-4. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 361 ................ 61

Figure 6-5. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 19 (Y-axis: from 0 to

1) ................................................................................... 6 1

Figure 6-6. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 19 (Y-axis: from 0 to

0 .2 ) .................................................................................................................. 6 2

Figure 6-7. Distribution of Facial Expressions Synchrony IDs from ID 20 to ID 38.....................62

Figure 6-8. Distribution of Facial Expressions Synchrony IDs from ID 39 to ID 57.....................63

Figure 6-9. Distribution of Facial Expressions Synchrony IDs from ID 97 to ID 115..................63

Figure 6-10. Distribution of the Sum of Facial Expressions Synchrony IDs from ID 1 to ID 19......64

Figure 6-11. Distribution of the Sum of Facial Expressions Synchrony IDs from ID 115 to ID

133 ........................................................... 64

Figure 6-12. Interaction satisfaction, Information satisfaction and Empathy score .................... 66

Figure 7-1. Ten Synchronized Facial Expressions that take up the largest percentage in the entire

interaction in each group .......................................................... .. 71

Figure 7-2. Ten most frequent synchronized facial expressions in each group ........................... 72

Figure 7-3. Ten Synchronized Facial Expressions that take up the largest percentage of the entire

interaction in each group .............................................................. 74


18

Figure 7-5. Ten Synchronized Facial Expressions that take up largest percentage in the entire

interaction in each group .......................................................... .................. 77


Figure 8-1. The examples of social, genuine, and greeting smiles ................................... 81

Figure 8-2. Cohen's Kappa Coefficient among three coders .......................... 82

Chapter 1

Introduction

Imagine you went out to a local restaurant for dinner. You would probably enjoy the dinner more

if the waitress is friendly, kind and smiling rather than salty and disgruntled. Hochschild(1983)

showed how service providers, especially flight attendants, have to manage their emotion and display

positive affect when they interact with their passengers. This work is the first notable work in the

emotional labor field of organization behavior area that has influenced other work thereafter such as

exploring the inner feelings of service providers (Rafaeli and Sutton, 1989) and selecting a service

provider who would get less psychologically stressed (Morris & Feldman, 1996). The methodology

in their research focused on examining whether a smiling service provider affected the customer to

feel better or whether smiles were contagious between the service provider and customer by looking

at whether both of them smiled or not. In this thesis, we examine how nonverbal communication

between a service provider and a customer affects the customers' perceived satisfaction. Among

nonverbal communication features such as facial expressions, eye contact, postures and body

20

gestures, we primarily analyze the dynamics of facial expressions between people in face-to-face

conversations. Specifically, we look at both the service provider's smiles and the customer's smiles

as well as customers' reported feeling about the service. Previous studies have investigated

behavioral mimicry in dyadic interaction by having human coders at the location of the experiment to

check whether a certain behavior, such as smiling, happened over the entire interaction (Tsai, 2001).

Other approaches have recorded the interactions and had human coders review the film and count the

number of times a behavior occurred (Chartrand & Bargh, 1999; Barsade, 2002). We propose

quantitatively more accurate ways of measuring mimicry behavior by measuring the number of times

when it occurred and the percentage of time it takes up in the entire interaction. In addition, we

explore empathetic communication not only through shared smile behavior but also with other

complementary facial expressions such as "caring-upset" and "smile-interested". Since emotion is

subjective and emotions such as caring are hard to recognize, we also have people label their videos

after the interaction and compare the labels of what they said they felt with the labels of what others

perceive in their video. With regard to the analysis of smiles, we classify smiles into three different

kinds depending on the intent and context in which the smile happened to elicit its authenticity.

Grandey(2005) classified authentic and inauthentic smiles by their intensity and facial muscles, but

we would rather take into account the intent and the context.

1.1 Background

During conversations people tend to mimic one another's facial and body gestures, such as

smiling together, nodding their heads in unison, or each putting their hand on their chin. Research has

shown that synchronized nonverbal cues can influence face-to-face communication in a positive

manner (Kendon, 1970). Many aspects of mimicry behavior have been studied by social scientists. In

21

Chartrand's work, he demonstrated the chameleon effect, showing that those participants that

interacted with a confederate who imitated the participant's behavior, compared with the case in

which the confederate did not imitate the participant, felt the interaction was more pleasing

(Chartrand & Bargh, 1999). Other research has also investigated the positive influence of mimicry

behavior in varying situations. In student-teacher interaction, the rapport between student and teacher

was stronger when the students were copying the teacher's behavior (LaFrance, 1982). In counselor-

client conversation clients preferred those counselors who mimicked the clients' expressions to those

who did not (Maurer & Tindall, 1983).

There are many arguments, and growing evidence, to account for human behavioral mimicry.

According to the common-coding theory (Prinz, 1997; Knoblich & Flach, 2003), the representations

of generated action are affected by the representations of perceived action and vice versa. Decety

claims that people have similar representations of action (Decety J & Sommerville, 2003) and that

people mimic the physical movements of one another because they are projecting the other person's

situation to their own (Decety, 2004). Empathy produces this "kinesthetic" imitation (Lipps, 1903),

which induces people to think that they are sharing similar affective states and experiences with their

conversation partners (Decety, 2004). The result is that people feel as if they "connect" with others,

which influences the building and sustaining of relationships with others (Chartrand, 2005). As a

result people create positive social and emotional qualities including affiliation and rapport by

unconsciously mimicking the physical movements and expressions of one another when they interact

(Chartrand, 2005).

Mimicry behavior can also be advantageous in establishing successful business relationships.

In a study of facial mimicry between a service provider and a customer, at a coffee shop, there was a

positive correlation between mutually similar facial expressions and positive customer

evaluation(Barger, 2006). Moreover, a waitress received higher tips when she mimicked the

customers by repeating the order that she was told (Baaren, 2003). In service-oriented businesses that

22

rely on face-to-face interaction, such as coffee shops, restaurants, banks or hotels, it is vital to

establish and maintain good relationships with the customers. Although research in business

management has shed light on the importance of employee-customer interactions, little has been

done with respect to the analysis of dyadic interactions that focus on the behavior, and subsequent

influences, of one person's actions on the other. Notable examples by Pugh and Tsai demonstrate that

positive affect, smiling, and engaging eye contact can positively influence the customer's experience

(Pugh 2001; Tsai 2001).

1.2 Overview

The remainder of this thesis is organized as follows: chapter two explains how the experiment is

designed to have a banker interact with a customer altering his facial expression; chapter three

elaborates on measures that we quantify to understand customer satisfaction and facial expressions;

chapter four and chapter five emphasize the perspective differences between banker and customer;

chapter six and chapter seven analyze customer satisfaction and dyadic communication; chapter eight

is one of the most interesting parts of this thesis that investigates different kinds of smiles; In chapter

nine, we conclude our findings.

Chapter 2

Experiment Design

2.1 General Description of the Experiment

The general design of the experimental interaction is that of a professional banker interacting

with a customer interested in learning about financial services (Figure 2-1). The banker provides two

kinds of financial services, which are similar to real world services provided at a retail branch. The

first service is to cash a $5 voucher from the customer participant as compensation for participating

in the study. This part is designed to simulate a cashing a check scenario. The participant was told up

front that they would get $10 for compensation, but the banker told them they would have to fill out

more paperwork after the study to get the rest of the money they were owed and could only get $5

now. This manipulation was made to instill a slightly negative state in the customer in order to

approximate more accurately the situation where a customer might be going to a real bank assistant

for help. After the experiment ended the participant received the rest of the money without additional

paperwork. The second service is to explain one of the financial services that a customer chose to

learn more about: Home Equity Line of Credit (HELOC), Individual Retirement Arrangement (IRA),

Certificate of Deposit (CD), mortgages, credit card, or student financial plans (529 plans & CDs).

This part is to simulate the situation in which bank customers ask questions and receive information

about the financial product they are interested in. There were two male bankers and one of them

interacted with forty-six participants, while the other interacted with forty-five participants. There is

one banker and one customer in each experiment and the experiment was nonreciprocal one-with-

many design (David, 2007) (Figure 2-1). The experiment was conducted in a room equipped with a

desk, two chairs, bank service advertising pamphlets and two cameras to make the appearance alike

to a personal banking service section at banks (Figure 2-2). One camera was used to record the

banker's facial expressions and the other was used to record the participant's facial expressions.

Customer 1, Netralfacialexpressions Customer46, Neutral facialexpressions

C toCrer 2, Empathetic facial eKpressions Customer 47, Empathetic facialepresions

Customer 3, Awayssmilin facial expressions Customer 48,Alwyssmilin facialexpressions

Customer 4, Neutral facial expressions Customer49 Neutrlfacial eprssions

Benlrl ICustomer,Empathetic facialexpressions Snied CustomerSO, Bnflaeticlear2si

Customer 6, Anws flig facial expressions Customer 51, Always smng facial expressiom

SCustomer O, A waymilins acial expresio

Customer 45, Always smiligfacial expressions Customer9l, Neutral facial pressions

Figure 2-1. nonreciprocal one-with-many design

.... .... .... .... ..

Figure 2-2. Setting where banker and customer interact

2.2 Participants - Hired Bankers

We hired two professional personal bankers, each with over two years of career experience as

a personal banker, to do what they usually do at work - explain financial services. During the hiring,

we asked them if they would be willing and able to manipulate the type of facial expressions

displayed during interaction with the customer. Each banker agreed to alter his facial expressions in

three different ways, following these exact instructions:

Manipulation 1 - Neutral facial expressions: Please try to sustain neutral facial expressions

regardless of the changes in the customer's facial expressions over the entire interaction.

Manipulation 2 - Always smiling: Please try to keep smiling regardless of the changes in the

customer's facial expressions over the entire interaction.

Manipulation 3 - Complementary facial expressions i.e., empathetic : Please try to

understand the customer's feeling and respond to it appropriately by smiling when the customer

seems to feel good, showing caring facial expressions when the customer expresses concern, showing

neutral facial expressions when you need to express that you are listening to the customer sincerely

and carefully, etc.

Throughout the experiment, the bankers interacted with the customer as they would normally

do in a banking setting aside from the expression manipulation. This included greeting a customer,

providing proper information, and thanking the customer for their time. The facial expressions of the

bankers were unobtrusively videotaped and audio-recorded from the moment they met and greeted

the customer to the end when the customer left the seat.

2.3 Participants - Customers

Thirty males and sixteen females (n=46) were recruited through flyers who were interested in

receiving information about different financial services. Before the experiment started, they were told

that their face and voice would be recorded as banks normally do for security reasons. However, they

were not told that their facial expressions would be analyzed. This was to prevent them being aware

of the purpose of the study. Afterward, they were told about the expressions and helped to label them.

2.4 Procedures

Prior to the participant entering the room the banker was told which expression manipulation

to conduct. The participant was then allowed into the experiment room where they would interact

with the banker and learn about specific financial services. At the end of the experimental interaction,

which took about 10 minutes, both the banker and participant filled out 9-point Likert scale surveys

evaluating the quality of the service based on the most comprehensive and popular instrument

SERVQUAL(Parasuraman, 1985 & 1988) and the attitude of the banker. While the banker and

participant completed the surveys the experimenter transferred the video recorded experimental

session to DVDs. After the banker and participant were finished with surveys they were asked to

label the video data for their facial expressions and emotions. After labeling their own video

information they labeled the videos containing the person they interacted with.

2.5 Facial Expressions Coding

In this study, the banker labeled his own video data and the participants also labeled their

own data. Then, the banker labeled the participant's video data and the participants labeled the

banker's video data. Lastly, human coders not involved in the study labeled both the bankers' and the

participants' data. In the interface of the labeling software that the banker and the participant used,

"VideoLAN-VLC media player" plays the DVD and the labeling software provides an entity to enter

the time when a certain facial expression was observed and seven emotion labels to select (Figure 2-

3). These seven labels are: smile, concerned, caring, confused, upset, sorry, and neutral. If there was

no proper label to choose from, the user could press "Other" and enter another label that they think is

appropriate for the expression. The labelers were instructed to stop playing the video and click on the

label button when they saw a facial expression, and then to continue to play the video until they saw

a change in the facial expression. On the right side of the user interface, there was a text box

displaying the time and the labeling result and it was editable so that the user could annotate the

reason for each facial expression, e.g. "smile - he made me laugh". By providing the space to type

the comment, we could learn more about the intent behind the facial expressions, especially the

smiles, where we would later categorize them into "Greeting Smile," "Social Smile," and "Genuine

28

Smile." For example, when the annotation was "smiling to greet the customer", it was classified as

"Greeting Smile" and when the annotation was "smiling to be polite to the customer", the smile was

classified as "Social Smile". "Genuine Smile" can be annotated as "hearing about getting the cash",

"glad to hear about tax deductions", "the banker told me a funny joke", etc. There were four human

coders not involved in the actual experiments. These labelers used an online video annotation

program, VidL, developed by the experimenters to label the video data (Figure 2-4). The interface

contains nine labeling buttons for facial expressions, which are composed of smile, concerned, caring,

confused, upset, neutral, satisfied, surprised and other and four labels for gestures including head nod,

headshake, chin on hand, open arms. The last three buttons are to record the points when the

participant started filling out a survey asking about a demographic profile at the very beginning of

the interaction and the last scene of the interaction since we could not observe the participants face

during this period.

Figure 2-3. Labeling interface used by banker and customer

Figure 2-4. Labeling interface used by four outside coders; The labels differ slightly here andthese data turned out to not be used in this thesis.

Chapter 3

Measures

3.1 Facial Expression Measures

3.1.1 Percent synchrony time of the facial expressions

We measured the duration of each of the facial expressions studied for both the banker and

participant. In total there are nineteen facial expressions that can be assigned by the human coder

(Table 3-1). Initially, we provided eight facial expressions labels shown in Figure 2-3 and the human

coders entered other emotion labels such as "Surprised", "Interested", "Annoyed" when they pressed

the button "Other". The labels in Table 1 include the seven labels provided in the labeling software

as well as the labels that came out of the "Other" category. Therefore, there are 19* 19 = 361 possible

pairs of synchronized facial expressions between a banker and participant and each synchrony is

assigned a unique identification. For example, "Banker: Smile - Customer: Smile" is assigned to

"Synchrony ID : 1" and "Banker: Concerned - Customer: Smile" is assigned to "Synchrony ID : 20".

In this context, synchronization means a pair of facial expressions between the banker and the

customer that overlaps in time. The percent synchrony time of the facial expressions is defined as the

percentage of the time each synchrony takes up in the entire interaction; the length of the synchrony

is divided by the entire interaction time length.

ID Facial Expressions ID Facial Expressions

1 Smile 11 Relieved

2 Concerned 12 Interested

3 Caring 13 Bored

4 Confused 14 Enthusiastic

5 Upset 15 Persuasive

6 Sorry 16 Annoyed

7 Neutral 17 Survey

8 Other 18 The end

9 Satisfied 19 Missing Label

10 Surprised

Table 3-1. Twenty-one labels for the facial expressions

3.1.2. Frequency of each facial expression

This measure counts how many times each facial expression happened over the entire

interaction.

3.2 Customer Satisfaction Measures

3.2.1. Interaction satisfaction

The participants answered the question, "How satisfied are you with the interaction overall?",

with a 9-point Likert scale rating.

3.2.2. Information satisfaction

The participants also evaluated information satisfaction with the question, "How satisfied are

you with the financial information provided?", using a 9-point Likert scale rating.

3.2.3. Empathy of the bankers

A survey was given to ascertain the customer's interpretation of the service provider's

empathetic attitude. The survey was based on the most commonly used instrument SERVQUAL

(Parasuraman, 1985 & 1988) that investigates five aspects of the service;"Tangibles, Reliability,

Responsiveness, Assurance, Empathy." The four items in the "Empathy" section of SERVQUAL

were adopted and the rating used a 9-point Likert scale.

3.2.4. Customer perception

The customers were asked to choose one of three attitudes for the banker: neutral, always

smiling, and empathetic. We measured this to see whether customer perception did or did not agree

with what the banker intended to convey.

Chapter 4

Banker Manipulation

4.1 Manipulation Checks

Figure 4-1 shows three labeled interactions illustrating interactions where the banker

successfully performed the facial expressions for each condition. In Figure 4-1(a), "Neutral," the

banker is showing neutral facial expressions most of the time, visualized with light blue color, and

other facial expressions such as smile and concerned are rarely observed. In Figure 4-1(b) "Always

Smiling", the banker is smiling throughout the interaction, which is visualized with orange bars. In

Figure 4-1(c), "Complementary facial expressions", we can see more dynamics in the banker's facial

expressions, i.e., transitions between different facial expressions, and we observe where the banker

and customer are smiling together. Additionally we see that the banker was responding with "caring"

facial expressions when the customer was expressing "confused" or "other".

- Smile

-ConcernedBanker

-- - Caring

-Confused

Customer Upset

- Sorry

Neutral

-Other0:00:48 0:01:52 0:02:57 0:04:02 0:05:07 0:06:12 0-07:16 0:08:21 0-09:26 0:10:31 0:11:36 0:12:40 0:13:45 0:14:50 Time(hr:mm:sec)

(a)Neutral manipulation

- Smile

-ConcernedBanker

- Caring

-Confused

Customer - Upset

- Sorry

Neutral

- Other0:00:26 0:01:09 0:01:2 0:02:36 0:05:19 0:04:02 0:04:45 0:05:28 0-06:12 006:55 0:07:38 0:08:21 0:09:04 Time(hr:mm:sec)

(b) Always Smiling manipulation

- Smile

- ConcernedBanker

.* m a a an e Ii - Caring

-Confused

Customer - Upset

- sorry

Neutral

-Other0:00:43 0:02:10 0:03:36 0:05:02 0:06:29 0:07:55 0:09:22 0:10:48 0:12:14 0:13:41 0:15:07 0:16:34 0:18:00 0:19:26 Time(hrmm:sec)

(c) Complementary facial expressions manipulation

Figure 4-1. Banker Manipulation Checks

,~:::::::::::~;;;;;;;~

We analyzed banker 2's data which has forty-six pairs of interactions in this section as well

as in the remainder of the thesis since banker 1's data still needs to be cleaned up in a right format.

Table 4-1 shows six quantitative measurements to distinguish three different manipulations. These

six measurements are as follows:

1. The percent of smiles is defined as the time during which the banker was smiling during

the entire interaction time. For example, in Figure 4-1 (c), the sum of each smile segment colored

with yellow bar is 3 minutes and 14 seconds and the total interaction time is 19 minutes at 33 seconds.

3 minutes 14 secondsTherefore, the percent of smiles is 3 minutes 14 seconds= 0.165 = 16.5%. The first row in Table 4-1

19 minsutes 33 seconds

summarizes the average of the percent of smile in 46 interactions for each manipulation.

2. The percent of neutral facial expressions is similar to the first measurement, the percent of

smiles, except that this measurement calculates how much percentage the neutral facial expressions

takes up in each interaction. The values in Table 4-1 are averaged over 46 interactions.

3. The number of banker transitions counts how many times the banker changed his facial

expressions from one to another. For example, in Figure 4-1(b) , the banker changes twice,

smile(yellow bar) -* concemed(red bar) - smile(yellow bar).

4. The difference between banker's transitions and customer's transitions is calculated by

subtracting the number of customer's facial expressions transitions from the number of banker

transitions. For example, in Figure 4-1(b), the number of banker's transitions is 3 and the number of

customer's transitions is 13. Therefore, the difference between the banker's transitions and

customer's transitions is 3-13 = -10 in this case. The negative value means the customer had more

transitions and was more dynamic. The positive value implies that the banker made more transitions.

If this value is zero, both the banker and the customer made the same number of transitions.

5. The number of smiles counts how many times the banker smiled in the interaction. If a

banker starts to smile and stops smiling and his facial expressions changes into another but smiling

then it is counted as one smile. For example, in Figure 4-1(b), the number of smiles is 2 and, in

Figure 4-1(c), the number of smiles is 11.

6. The number of neutral facial expressions is similar to the number of smiles except that it

counts how many times the banker showed neutral facial expressions.

As we can see from Table 4-1, the banker was smiling most of the time throughout the

"always smiling" manipulation, which was 96.18% of the interaction time on average, while he was

smiling only 1.01% of the interaction time in "neutral" manipulation. The banker was also

successfully following the manipulation instruction by maintaining neutral facial expressions 98.75%

of the interaction time.

Banker ComplementaryManipulation Neutral Manipulation Always Smiling Facial ExpressionsNeutral Manipulation Manipulation

Measurement Manipulation

The percent of smile 1.0% 96.2% 25.6%

The percent of Neutral 98.8% 3.7% 68.0%Facial Expressions

The number of banker 0.8 0.9 13.4transitions

The difference betweenbanker's transitions and -14.3 -12.9 -4.3customer's transitions

The number of smiles 0.5 1.4 6.6

The number of neutral 1.3 0.4 6facial expressions

Table 4-1. Banker smiled most of the time in the "Always Smiling" manipulation, showedneutral facial expressions for 98.75% of the interaction time in "Neutral" manipulation and was

more dynamic in "Complementary Facial Expressions" manipulation.

Figure 4-2 shows the histogram of these six measurements. From the histogram of Measurement

3,the number of banker transitions, we can see that the banker made more transitions during the

complementary facial expressions manipulation averaging 13.4, which is much more dynamic than

............

the other two groups averaging close to 0.8. In addition, the dynamics between the banker and the

customer tend to become similar in "Complementary Facial Expressions" manipulations. We can

confirm this tendency from the histogram of Measurement 4,"The Difference Between Banker

Transitions and Customer Transitions" since their difference was lower in "Complementary Facial

Expressions" manipulations averaging -4.3 compared to the other two cases that were -14.3 and -12.9.

Neutral Manipulation (N=15)

0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100 (%)

(a) Neutral Manipulation

Always Smiling Manipulation (N=16)

0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100 (%)

(b) Always Smiling Manipulation

Empathetic Manipulation (N=15)

0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100 (%)

(c) Empathetic Manipulation

Figure 4-2. Histogram of Measurement 1.The percent of smile

16

14

S 12

10

80

6

4

0

Neutral Manipulation (N=15)16

14

S 12

A 86

2

0

14

12

0 10

8

0o 6

2

0

0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100 (%)

(a) Neutral Manipulation


0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100 (%)


Empathetic Manipulation (N=15)4.5

2.5

. 2

.o 1.5

0.5

0

0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100 (%)

(c)Empathetic Manipulation

Figure 4-3. Histogram of Measurement 2.The percent of neutral facial expressions

-I

Neutral Manipulation (N=15)14

, 12

S 2o0-2 3-5 6-8 9-11 12-14 15-17 18-20 21-23

The number of banker's transitions

(a)Neutral Manipulation

Always Smiling Manipulation (N=16)16

14

12

a 10

8

4

2o

0-2 3-5 6-8 9-11 12-14 15-17 18-20 21-23




l"

4.5

4

3.53

2.5

S 2S1.5

1

0.5n

0-2 3-5 6-8 9-11 12-14 15-17 18-20 21-23



Figure 4-4. Histogram of Measurement 3.The number of banker's transitions

-I

Neutral Manipulation (N=15)

6

5

4

3

2

0

(-)50-55 (-)45-49 (-)40--44 (-)35-39 (-)30-34 (-)25-29 (-)20-24 (-)15-19 (-)10-14 (-)5-9 0-(-)4 (+)1-5 (+)6-10 (+)11-15

The difference between banker's transitions and customer's transitions

(a)Neutral Manipulation


6

5

o 4

3

20

(-)50-55 (-)45-49 (-)40-44 (-)35-39 (-)30-34 (-)25-29 (-)20-24 (-)15-19 (-)10-14 (-)5-9 0-(-)4 (+)1-5 (+)6-10 (+)11-15


(b)Always Smiling Manipulation


7

6

5

3

2

(-)50-55 (-)45-49 (-)40-44 (-)35-39 (-)30-34 (-)25-29 (-)20-24 (-)15-19 (-)10-14 (-)5-9 0-(-)4 (+)1-5 (+)6-10 (+)11-15



Figure 4-5. Histogram of Measurement 4.The difference between banker's transitions and customer'stransitions

I

r - ..... ....................... ........... ................. ........... ......................

I---- - - --- -- - -- -I I ----

I I I I I I--t--~-I I I I

0.8 ------ --- ---r--- --------- N

* Empathetic

* Ahvays Smilng0.2----- - ------ - -I- - - -- - -)-- -- - Mean of Neutral

SMean of Empathetic

I I Mean of Always Smiling

0 1 2 3 4 5 6 7 8 9 10

The number of smile

Figure 4-6. The number of smiles and the percent of smiles

I I I I I I

I II I I I I

S0.8------ --- ------------ --- --------

- 0.6 ------- ----- - -

0 0.4------0---- Neutral

SI I I Empathetic

I I Always Smiling0.2 ----- -- - -- -- - -.- * Mean of Neutral

A i i | I I * MeanofEmpatheticA Mean of Always Smiling

0 1 2 3 4 5 6 7 8 9 10

The number of neutral facial expressions

Figure 4-7. The number of neutral facial expressions and the percent of neutral facial expressions

Measurements 5 and 6 can be observed more clearly when we combine these measures with the

percentage measurements, i.e., measurement 1 and measurement 2 (Figure 4-6 & Figure 4-7). That is,

there are a smaller number of smiles or neutral facial expressions in "always smiling" and "neutral

-~

facial expressions" manipulations because the banker was told to keep the same facial expressions,

while the percent is high because the banker was trying to keep the same facial expressions all the

time. Meanwhile, in the "complementary facial expressions" manipulation, there are more smiles or

neutral facial expressions, but the percentage is not the highest one among the groups. From this

measurement, we can confirm there are three distinct forms of manipulations taking place.

4.2 Banker and Customer Perspective Differences

Interestingly, despite the differences between the three facial manipulations acted by the

banker (Table 4-1), a customer's perception of the banker's attitude did not always agree with a

banker's manipulation.

ustomer Perception

Neutral Always Smiling EmpatheticBanker Manipulation

7 6 2Neutral - = 0.5 - = 0.4 = 0.1

15 15 15

2 8 6Always Smiling - = 0.1 - = 0.5 - = 0.4

16 16 16

Complementary 3 5 7Facial Expressions 15 15 15

Table 4-2. Banker manipulation vs. customer perception

Table 4-2 shows that when the banker tried to maintain a smile throughout the interaction,

only half of sixteen customers reported that they thought the banker was always smiling, while two

of sixteen customers thought the banker's attitude was neutral. Surprisingly, six out of fifteen

customers in the neutral manipulation sessions reported that they perceived the bankers attitude as

"Always Smiling". This may be due to voice tone and the typical notion that bankers usually smile at

their customers. Meanwhile, the probability that the customer would feel the banker was empathetic

was highest when the banker was trying to respond with complementary facial expressions and the

probability of the customer perception to be "Always Smiling" was highest when the banker intended

to always smile. Since a customer's perception does not always match a banker's intent, we needed

something more objective for our analysis. We therefore conducted three kinds of analysis,

classifying the data into three conditions, i.e., neutral, always smiling, empathetic (complementary

facial expressions), according to three different measures: the cognitively reported customer

perception, a banker's intent, and the measured synchrony labeled using their facial expressions. We

present these three analyses in chapter six and chapter seven after discussing the differences in

perspective in chapter five.

Chapter 5

First Perspective Labeling and Second

Perspective Labeling

After filling out the survey asking about their interaction satisfaction, both the banker and the

customer reviewed their own video data recorded during their interaction. Firstly, they labeled their

own facial expressions from the DVD that recorded them. Then the experimenter switched DVDs so

that the banker could label the customer's facial expressions and the customer could label the

banker's facial expressions. The first perspective labeling refers to the facial expressions labeling that

was done while each person was watching their own DVD and the second perspective labeling

corresponds to the facial expressions labeling that was done while they were watching their partner's

DVD. In sections 5.1 and 5.2, we compare these two different perspectives to observe the facial

expressions and list which facial expressions tend to have relatively more agreement between these

two perspectives and which facial expressions are ambiguous.

5.1 Data Format

Since both the banker and the customer reviewed their own DVD and their partner's DVD, there

were four labeling files in each experiment; banker's labeling of his own DVD, banker's labeling of

the customer's DVD, customer's labeling of his/her own DVD and customer's labeling of the

banker's DVD. Figure 5-1, Figure 5-2, Figure 5-3 and Figure 5-4 show one data sample with these

four labeling results. Figure 5-1 is the banker's labeling of his own DVD. The first line starts with the

time when the banker meets and greets the customer and the last line is the time when the banker

finishes the conversation and the customer leaves the room. In this sample, the banker labeled that he

was smiling when he was greeting to meet the customer and recapitulating his explanation session.

The first line and the last line have to be the same in all four files, which means their interaction time

is equal. In Figure 5-2, Figure 5-3 and Figure 5-4, the time of the first line is the same as the one in

Figure 5-1, yet the label is "Missing Label", which means the coder did not start labeling at that point.

In addition, the coders annotated the reason why they were making a certain facial expression at that

moment. We make use of these comments to understand the intent and context of the facial

expressions and classify smiles into three categories in chapter eight. By comparing the banker's

labeling in Figure 5-1 with the customer's labeling of banker's DVD in Figure 5-2, we notice that

there exists some agreement in smile labeling and there exists some disagreements in "Neutral" and

"Caring" facial expressions. In the following section, we calculate the agreement between banker's

labeling and customer's labeling for each facial expression.

Figure 5-1. Banker's Labeling of Banker's DVD

Figure 5-2. Customer's Labeling of Banker's DVD

........ ..............

Figure 5-3. Customer's Labeling of Customer's DVD

Figure 5-4. Banker's Labeling of Customer's DVD

5.2 Agreement between First and Second Perspective

Labeling

We have used three measurements to compare first and second perspective labeling. In all three

measurements, the numerator is the length of the time during which the banker's labeling and the

customer labeling agree. The difference between these three measurements is that measurement 1

compares the agreement between the banker and the customer with banker's labeling, measurement 2

compares it with customer's labeling, and measurement 3 compares it with the length of the

interaction. To explain these three measurements specifically, let's take an example with the data

shown in Figure 5-1, Figure 5-2, Figure 5-3 and Figure 5-4 of section 5.1 above.

Banker's Labeling Duration Customer's Labeling Duration

6:46:42 Smile 5 seconds 6:46:42 Missing Label 2 seconds6:46:47 Smile 24 seconds 6:46:44 Smile 1 seconds6:47:11 Smile 14 seconds . 6:46:55 Smile 12 seconds6:47:25 Smile 7 seconds 6:47:07 Caring 15 seconds6:47:32 Smile 20 seconds 6:47:22 Smile 5 seconds6:47:52 Smile 11 seconds 6:47:27 Smile 17 seconds6:48:03 Smile 23 seconds 6:47:44 Neutral 8 seconds6:48:26 Smile 49 seconds 6:47:52 Concerned 5 seconds6:49:15 Smile 1 minutes 4 seconds 6:47:57 Smile 27 seconds6:50:19 Smile 14 seconds 6:48:24 Smile 10 seconds6:50:33 Smile 50 seconds 6:48:34 Concerned 6 seconds6:51:23 Smile 7 seconds 6:48:40 Smile 6 seconds6:51:30 Smile 29 seconds 6:48:46 Neutral 3 seconds6:51:59 Smile 6 seconds 6:48:49 Smile 6 seconds6:52:05 End of the interaction 6:48:55 Concerned 20 seconds

6:49:15 Smile 8 seconds6:49:23 Concerned 6 seconds6:49:29 Smile 15 seconds6:49:44 Caring 33 seconds6:50:17 Smile 7 seconds6:50:24 Caring 10 seconds6:50:34 Smile 6 seconds6:50:40 Caring 28 seconds6:51:08 Smile 4 seconds6:51:12 Caring 13 seconds6:51:25 Smile 9 seconds6:51:34 Caring 26 seconds6:52:00 Smile 5 seconds6:52:05 End of interaction

Figure 5-5. Banker's Labeling of Banker's DVD and Customer's Labeling of Banker's DVD

Figure 5-5 compares the banker's labeling of the banker's DVD, which is the same as Figure 5-1,

with the customer's labeling of the banker's DVD, which is the same as Figure 5-2. As we can see on

the left side of Figure 5-5, according to the banker's self labeling the banker was smiling during the

entire interaction time, 5minutes 23 seconds. Meanwhile, according to the customer's labeling the

banker was smiling only 2 minutes and 23 seconds, which is the summation of all of the smiling

segments(11 + 12 + 5 + 17 + 27 + 10 + 6 + 6 + 8 + 15 + 7 + 6 + 4 + 9 seconds = 143 seconds = 2

minutes and 23 seconds). In this case, the value of measurement 1 of smiles is 0.443, which can be

calculated like the following:

The duration of banker and customer labeling agreement of smile 2 minutes 23 seconds 143seconds- -= 0.443

The duration of banker's labeling of smile 5minutes 23 seconds 323seconds

Therefore, measurement 1 is the ratio between the banker and customer labeling agreement and

banker's labeling. Measurement 2 can be calculated similar to measurement 1 except that the

denominator is the duration of the customer's labeling of smile this time. Measurement 2 can be

calculated like the following:

The duration of banker and customer labeling agreement of smile 2 minutes 23 seconds 143seconds

The duration of customer's labeling of smile 2minutes 23 seconds 143seconds

Measurement 3 is used to calculate how much ratio the banker and customer labeling agreement of

smile takes up during the entire interaction time.

The duration of banker and customer labeling agreement of smile_ 2 minutes 23 seconds_ 143seconds

The Interaction Time 5minutes 23 seconds 323seconds

Table 5-1 shows these three measurements for all the facial expressions IDs, from ID 1, smile, to ID

19, missing label. The values listed here are the average of forty-six data of banker2 in the

experiment. For example, if we calculate measurement 1 for facial expressions ID 2= concerned =

with the data provided in Figure 5-5, the value was ignored when the average was calculated since

the duration of the banker's labelingof concerned is zero in this data. Table 5-1 shows the

comparison between the banker's self labeling of the banker's DVD and the customer's labeling of

the banker's DVD, which means the banker's labeling is the first person perspective labeling and the

customer's labeling is the second person perspective labeling. As we can see from Table 5-1, "Facial

Expressions ID 1: Smile" made a fairly high ratio of agreement between first person and second

person perspectives labeling, which takes up 45.2% among the banker's smile labeling and 43.4%

among the customer's labeling. "Facial Expressions ID 7: Neutral" was also relatively higher than

others, which was 43.3% among the banker's labeling and 67.6% of the customer's labeling.

Measurement Measurementl. Measurement 2. Measurement 3.Banker and Customer agreement Banker and Customer agreement Banker and Customer agreement

Banker's Labeling Customer's Labeling Interaction Time

Facial Expressions ID

1 : Smile 0.452 0.434 0.152

2 : Concerned 0 0 0

3 : Caring 0.222 0.011 0.003

4: Confused 0 0 0

5 : Upset Not Applicable 0 0

6: Sorry 0 0 0

7 : Neutral 0.433 0.676 0.282

8 : Other 0 0 0

9 : Satisfied Not Applicable 0 0

10 : Surprised Not Applicable 0 0

11 : Relieved Not Applicable 0 0

12 : Interested Not Applicable 0 0

13 : Bored Not Applicable 0 0

14 : Enthusiastic Not Applicable 0 0

15 : Persuasive, Convincing Not Applicable 0 0

16 : Annoyed Not Applicable 0 0

17 : Survey Not Applicable Not Applicable Not Applicable

18 : The end of interaction 0 0 0

19 : Missing Label 0 0 0

Table 5-1. Comparison between banker self labeling and the customer's labeling of banker'sDVD for each facial expression ID; banker's self labeling is the first person perspective labeling

and customer's labeling is the second person perspective labeling

Table 5-2 shows the same three measurements for the customer's self labeling of the customer's

DVD and the banker's labeling of customer's DVD. Since the customer's labeling is the first person

perspective labeling and the banker's labeling is the second person perspective labeling in this case,

the denominator of measurement 1 is the duration of the customer's labeling of each facial

expression and the denominator of measurement 2 is the duration of the banker's labeling of each

facial expression. As is shown in Table 5-2, "Facial Expressions ID 7: Neutral" had the highest

agreement, which agreed with 89.9% of customer's self labeling and 52.7% of the banker's labeling

of customer's facial expressions.

Measurement Measurementl. Measurement 2. Measurement 3.Banker and Customer agreement Banker and Customer agreement Banker and Customer agreement

Customer's Labeling Banker's Labeling Interaction Time

Facial Expressions ID

1: Smile 0.311 0.510 0.038

2 : Concerned 0.014 0.199 0.003

3 : Caring 0 0 0

4 Confused 0.033 0.172 0.001

5 : Upset 0 0 0

6: Sorry 0 0 0

7 : Neutral 0.899 0.527 0.410

8 : Other 0.064 0.286 0.001

9 : Satisfied Not Applicable Not Applicable Not Applicable

10 : Surprised 0 0 0

11 :Relieved 0 0 0

12 : Interested 0 0 0

13 : Bored 0 0 0

14 : Enthusiastic 0 0 0

15 : Persuasive, Convincing 0 0 0

16: Annoyed 0 0 0

17 : Survey 0.8592 0.765 0.098

18 : The end of interaction 0 0 0

19: Missing Label 0.3366 0.643 0.005

Table 5-2. Comparison between customer self labeling and the banker's labeling of customer'sDVD for each facial expression ID; the customer's self labeling is the first person perspective

labeling and the banker's labeling is the second person perspective labeling

"Facial Expressions ID 1: Smile" also showed more than 50% agreement in the banker's labeling and

31.1% of customer's labeling. We can conclude from the results in Table 5-1 and Table 5-2, smiles

and neutral facial expressions were quite obviously recognizable between different coders, while

caring, concerned, and confused were less obvious in comparison. More interestingly, other facial

expressions except for these ones i.e., smile, neutral, caring, concerned and confused, were very

dependent on the coder's perspective. Therefore, to analyze and understand dyadic communication

not only through shared smiles but also with other complementary facial expressions pairs such as

"Smile-Satisfied", "Caring-Upset", and "Caring-Bored", we adopt both the first person perspective

labeling and the second person perspective labeling in chapters six and seven.

Chapter 6

Customer Satisfaction Data Analyses

6.1 Hypotheses

H1. Customers are more satisfied with the interaction and the information when a customer

feels the service provider is empathetic or always smiling.

H2. There are three distinctive and valuable ways to investigate customer satisfaction and

these are what the service provider is trying to do, what the customer perceives and what actually

happens in their interaction.

The remainder of this chapter analyzes the customer satisfaction data according to

cognitively reported customer perception, banker manipulation and affectively measured

synchronized facial expressions.

6.2 Analysis 1. Cognitively Reported Customer Perception

Interaction satisfaction and information satisfaction were highest in the "Empathetic" group

and SERVQUAL empathy score of a service provider measured with surveys was highest in the

"Always Smiling" group, yet these two groups showed similar mean of ratings in all of these three

measurements (Figure 6-1). An ANOVA was run on each of Interaction Satisfaction, Information

Satisfaction and SERVQUAL empathy score. In each ANOVA, "Interaction Satisfaction" (F =

12.855, p = 0.000042) and "Information Satisfaction" (F = 7.036, p = 0.0023) and "Empathy

Score"(F = 8.672, p = 0.0007) were significantly lower in the "Neutral" group than "Empathetic" and

"Always Smiling" group.

I Neutral Always Smiling U Empathetic

NeutralInteraction : Mean = 6.083, SD = 1.165Information : Mean = 6.667, SD = 1.371Empathy : Mean = 4.958, SD = 1.339

Always SmilingInteraction : Mean = 7.737, SD = 1.147Information : Mean = 7.947, SO = 1.079Empathy : Mean = 6.868, SD = 1.311

EmpatheticInteraction : Mean 8.067, SD = 0.884Information : Mean = 8.067, SD = 0.704Empathy : Mean = 6.7833, SD = 1.379

ANOVAInteraction : F = 12855, p = 4.2E-05Information : F = 7.036, p = 0.0023Empathy : F = 8.672, p = 0.0007

Interacton Satisfacton Information Satisfaction EmpathyScore

Figure 6-1. Interaction Satisfaction, Information Satisfaction and Empathy Score

56

:~""""""""""""""""~

6.3 Analysis 2. Banker Manipulation

While there are obvious differences among the three measures, Interaction Satisfaction,

Information Satisfaction, and SERVQUAL Empathy Score, when the data is classified according to

customer perception, these measures are similar when we classify the data according to the banker's

manipulation. As it is shown in Figure 6-2, Interaction Satisfaction, Information Satisfaction and

Empathy Score scored highest in "Always Smiling" group (when the banker tried to always smile)

and scored lowest in "Neutral" group(when the banker appeared neutral). However, all of the P-

values, p(Interaction, Information, Empathy Score) = (0.126, 0.287, 0.424), are higher than 0.05 in

ANOVA, which indicates that the difference is not statistically significant.

1s M Neutral

* Akamys Smiling

- Cnmprenwntary Firial Fressinni

| - NetralInteractior : Mean = 7, SD = 1254Inbrmetioi : Mear = 7.267, SD = 1.163Epathy : Mean = 6.05, SD = 1.203

Always Sm lingInteractior : Mean = 7.938, SD = 0.9985 Infortio : Mea = 7.938, SD = 1.1814 lpaty : Mean =b.1, W = 1./ZZ

Interactior : Mean = 7.267, SD = 1.580nfrrrtio : Memn = 7.733, SD = 1.223

2 eIpathy : Mean = 6.2, SD= 1678

1 ANOVAInteractior : F= 2.178, p = 0.126nr h rmtlon : F = 1.284, p = 0287

Interacion satlstactlOu Information atisfaction Epathyscore Lmpathy : 1 = U.8 /b, p = 0.424

Figure 6-2. Interaction Satisfaction, Information Satisfaction, Empathy Score

r. ............ ........

6.4 Analysis 3. Affectively Measured Synchronized Facial

Expressions

We classified the data into three groups by directly measuring the synchronized facial expressions: If

there were more than 10% of shared smiles in the interaction, then the data was placed into the

"Empathetic" group. We also classified not only smile pairs but also other complementary facial

expressions pairs in which the banker was expressing "caring" and the customer was "upset", the

banker was smiling while the customer was satisfied or interested as part of the "Empathetic" group.

With regard to the "Banker : caring - Customer : upset" pair, "Banker : Smile - Customer : Satisfied"

pair, and "Banker : Smile - Customer : Interested" pair, if the percent of one of these pairs is more

than 0%; then the data was classified as part of the "Empathetic" category. These criteria were

chosen by looking at the quantitatively measured percentage of each synchrony pair and comparing

this measurement among the data set. As we can see from Figure 6-6, there is a distinct boundary at

10% in "Banker: smile - Customer: smile" synchrony, which means the data with 10% or more

shared smiles have a relatively large percentage of shared smiles compared to other data in this data

set. If there were fewer than 10% of shared smiles throughout the interaction and more than 40% of

the time the banker smiled, then the data were classified into the "Always Smiling" group. There are

more rules to classify the data into three groups: neutral, always smiling and empathetic. These rules

are explained in Figure 6-3 below.

An interaction ->data

no

no

Ino

Empathetic

> Neutralyes

no

Always Smiling

Always Smiling

Figure 6-3. Classification Rules

Figures 6-4 to 6-6 plot the distribution of each synchrony. In these figures, the x-axis corresponds to

the 361 facial expressions synchrony IDs listed in Appendix A and the y-axis indicates the

percentage of each synchrony in each interaction. Therefore, Figure 6-5 shows the distribution of 361

facial expressions synchrony IDs of the forty-six pairs of interactions of Banker 2. The decision

boundaries in the classification rules above (Figure 6-3) were chosen by hand based on the

distribution of these facial expressions synchronies. We can see a lot of data at the range from

Synchrony ID 1 to Synchrony ID 50 and at the range from Synchrony ID 100 to Synchrony ID 150 in

Figure 6-4. To observe this region in detail, the range from Synchrony ID 1 to Synchrony ID 19 is

plotted in Figure 6-5. Synchrony ID 1 is the smile pair i.e., Banker: Smile - Customer: Smile and it is

less than 40% in general. Therefore, we scaled down the y-axis to take a look at the data closely,

which is redrawn in Figure 6-6. The red line in Figure 6-6 indicates there is a gap between the data

. .. .......

above the red line and the data below the red line, which means the red line could possibly be a

boundary to divide the data into two groups. That is how the first classification rule is taken in Figure

6-3. There is another red line on Synchrony ID 12 (Banker : Smile - Customer : Interested) in Figure

6-3. This synchrony is chosen to classify the data into the empathetic group if the data does not meet

the criteria to belong to the always smiling group, which has more than 40% of banker's smiles

without customer's shared smiles. Synchrony ID 12 can also be the banker's one-sided smile but it is

different from others in such that this synchrony is a positive one whereas others such as Synchrony

ID 2 (Banker : Smile - Customer : Concerned), Synchrony ID 4 (Banker : Smile - Customer :

Confused) or Synchrony ID 5 (Banker : Smile - Customer : Upset) can be considered as a negative

pair and a lack of empathy. There is one more positive pair which is the not shared smile pair and it

is Synchrony 9 (Banker: smile - Customer - satisfied). However, as is shown in Figure 6-6, this pair

was not found among the dyadic data so that criterion was not chosen to select the empathetic group.

There were rarely found data between Synchrony ID 20 (Banker: concerned - Customer : smile) and

Synchrony ID 38 (Banker: concerned - Customer : missing label) having only two points (Figure 6-7).

1 ------ -------- -------- ------- -------- -------- --------

0.6 ------ --- --- - - - ------ ------- - - - -- - - -I-

-

0.8

Facial Expressions Synchrony ID0.8 ------ r --- I---I- -- --- ----- r------ I----- -----------

0 6 - - - - ---- - -, 0.4- --- -- I-- -- ----- -I---

0'

0.2 --- c- - -- -- --- --0 2 100 150 200 250 300 350

Facial Expressions Synchrony ID

Figure 6-4. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 361

I -- t----- I- -"------- L- -- - -- -

,x 0.8 --- T-T----------------------------------------- --------I I * T I,

; 0.6 I III

o I I

0.4 ----- ------- --------------------- -- --- -- -I

0.2 1 I I__ ........ i

0

0 2 4 6 8 10 12 14 16 18 20


Figure 6-5. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 19 (Y-axis: from 0 to 1)

----I I I

-- 1- ----- I-- -- I-I I II I I

---- --------- -----1---- ---------- ;--- - ---

-- ---------.... .-

i I II I I

I I I

i I I

I II I -

- -- i

- - - I

III I1 I

----------- ----- - -- --

-t----- -------- ---- ----

--- -------

--------

--- --- ---- -, , _ __ _

r ---

I I~3

0.2

0.18

0.16

0.14

0.12

0.1

0.08

0.06

0.04

0.02

0


Figure 6-6. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 19 (Y-axis: from 0 to 0.2)

0.9 -- - -------- ___--- --

0.8 -- - ---- ---- - - - - - - - - - - - - - - - -- - - - - T - - - - - -- - - - - - - - - - - -0.7 i-- ------------- ---- ----- ---- ----- ---- --- I--

0. -- - - -- - - - - - - - - -- - - - - ----- - - - - - - -

0.6 - -------- ----- --- --------- ----- ------ ---------- --- -

0 0.5 - -- - - -- --t -----t -- ---- -- --- -- -----r -- ---- -- -I -

0.2 ----- --- ------ ------ ----- --- -- ----------- ---


Figure 6-7. Distribution of Facial Expressions Synchrony ID from ID 20 to ID 38

62

0 2 4 6 8 10 12 14 16 18 20

SI I I I I-----r-----------------------

0.9 - - - - - - - L - - - -

0. - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - -I I I I I I

0 0.5------- ----- ---- - -- 4- 1--- --.- ...

0 0.3 ------------------------------- ----------------------0.7 ---- --- -- - - ------- . - - - -,

0.1 ----- --- ------------

0.5

38 40 42 44 46 48 50 52 54 56 58

------ ----- ----- ------------------------ -- --- ---------

0.9 ------ ----- ----- r----- ----- ----- ------------------ - - - - --0 .8- -- -- - r - - - - -t - - - - -- T - - - - -T - - - - - - - - -j - - - - - - - - - - - - - - - -

S .8 - - - -- ---- ------ --------

0.3 ---------------- -- T ----- - - - - - - - - - - - - - - - - -- - - - - - -,0.4 ---- --- - ---------- - - - - ---- ------------ - --- --0.1-- - - --

0.1 ------ ------ L L ---- I ----- I ----- ----- i ----- - ------ -- ----0.2 ---------------- +--------------- ------------------ --

0.1 ------------- - ----- *----- ----- ------------ ----- --

0 +96 98 100 102 104 106 108 110 112 114


Figure 6-98. Distribution of Facial Expressions Synchrony IDs from ID 97 to ID 115

630.8----r------------------

o .4-----------+ -'- - -I... ... '

ni i i i i I1

63

--

1---------------- ------ --------- ---------------- -----I I II I I I

0.9 ----- - - -- -- ---- - --- --- -- -- -- --

0. -- -- -

0.7S0. I - - i - - - - - - - - -- - - - - - - - - - I

I I I I I

0.6 -- ---- --- ----- ----J -L---

0.8 -- - ------ - ------ --- ----

0.2-- I I +

0.3 t------ T------I--~ -- T ---- r---- --- -- r----- I ---- T ---- I

00 2 4 6 8 10 12 14 16 18 20


Figure 6-10. Distribution of the Sum of Facial Expressions Synchrony IDs from ID 1 to ID 19

0. ----- -------- ----- ------------------ ----I I I j I I I I I I

0.8---- - - -- - ---- - - --- - ----- ----0.5 - - ---- - -....- ,------, - - - , -

0. -- - - - I- ----- --- -- - -1 ----- ---- ---- 1

0) -- -- - - --- -- - - ------- -- -- I - - ----I I I I i I I I0.2 _

0.1 - --- -- - - - --- ------ ......

114 116 118 120 122 124 126 128 130 132 134


Figure 6-11. Distribution of the Sum of Facial Expressions Synchrony IDs from ID 115 to ID 133

64S 0.4 _I_ ----..----..------ ,--64

IIIIIIIIIIIIIIII - 111- nk.

Synchrony ID 43 (Banker : Caring - Customer : Upset) was also chosen to place the data into the

empathetic category as it is a complementary facial expression pair. The boundary value of this

synchrony is marked as 0% in Figure 6-3. There were seldom data between facial expressions

synchrony ID 96 and ID 115 (Figure 6-4). The x-axes in Figure 6-5 and Figure 6-6 have different

meanings from the ones in Figures 5-1, 5-2, 5-3, 5-4 and 6-3. Figure 6-5 plots the sum of banker-only

smiles without the customer returning a smile. The x-axis in Figure 6-5 means the synchrony ID

takes up the most portion of the sum. For example, if the sum is 0.44 and Synchrony ID 7 takes up

0.38 and the sum of other Synchrony IDs between Synchrony ID 2 and Synchrony ID 19 accounts

for the remaining 0.06%, then Synchrony ID 7 is the x-coordinate of the data and their sum 0.44 is

the y-coordinate of the data. The x-axis and y-axis in Figure 6-6 also have the same concept except

that neutral facial expressions are counted in this case. When we classified the data into neutral,

always smiling and empathetic groups following the classification rules in Figure 6-3, the number of

data in each group was quite evenly distributed, assigning fifteen to neutral, sixteen to empathetic

and fifteen to the always smiling group. Figure 6-12 shows the customer satisfaction analysis results

when the data was classified along with these affectively observed rules. The major difference

between Analysis 1 and Analysis 3 is that the Empathy Score is the highest in "Empathetic" group in

Analysis 3 i.e., affectively measured classification, whereas the Empathy Score is the highest in

"Always Smiling" group in Analysis 1 i.e., cognitively reported classification. In addition, in terms

of interaction satisfaction, the mean of the ratings of "Always Smiling" was close to the mean of the

"Empathetic" group in Analysis 1 yet, the mean of the ratings of "Always Smiling" was close to the

mean of the "Neutral" group in Analysis 3. Meanwhile, the "Empathetic" group scored highest in

"Information Satisfaction" in both analyses. However, in Analysis 3 the significance is not enough to

make strong differences between the three groups, which have p-values larger than 0.05 in ANOVA.

As is the result in this section, since there is a difference between the cognitively reported way to

investigate customer satisfaction and the affectively observed way, we need to analyze the customer

65

satisfaction data with both methods. Moreover, there might be more than one way to classify the data

according to the affectively measured method depending on how we describe the criteria. For

example, Synchrony ID 42 (Banker : Caring- Customer : Confused) could have been another

criterion to classify the data into empathetic group. Therefore more ways to select those criteria need

to be explored.

lOSNeutral I Always Smiling MEmpathetic

9 NeutralInteractio : Mean = 7.2, SD = 1.082

8 Infomrnian : Mean =7533, SD= 0.990

7 Empathy : Mean = 6217, SD = 1.168

Always Smlng6 Interaction : Mean = 7188, SD = 1.797

Infortin : Mean = 7.6, SD = 1.5495 Empathy : Mean= 6.183, SD= 2.145

4 "EmpathetIInteracti : Mean = 7867,D = 0.834

3 Maformwtm : Mean = 7813, SD= 1.047Empathy: Mean = 6.609, SD = 1242

2ANOVA

1 nteructio- : F = 1.321, p= 0278Irnatn :F = 0.224, p = 0.800

0 - pathy : F = 0354, p = 0.704

Inte radon Satisfaction Information Satisfaction Empathy Score

Figure 6-12. Interaction Satisfaction, Information Satisfaction and Empathy Score

6.5 Summary

From the result in Analysis 1, we could confirm hypothesis 1 that claims the customer

satisfaction is the best when the customer think the service provider is empathetic. This is a typical

and traditional way to investigate customer satisfaction. We suggested two more ways to analyze

customer satisfaction and these are to classify the data according to what the service provider tried to

convey and what actually happened in their nonverbal communication. When the data were classified

according to the banker's manipulation the customer satisfaction scored highest in the "Always

Smiling" manipulation in Analysis 2 and when the data were classified according to their detected

66

.. ... .

facial expressions in Analysis 3 the customer satisfaction was the highest in the empathetic group,

while in the other two groups i.e., Neutral and Always Smiling groups, the ratings were similar. This

result agrees with the result mentioned in Chapter 4 that people's intent and perception is subjective.

Therefore, these three methods are valuable, and should be furthur investigated and kept individually

to complement each other.

Chapter 7

Dyadic Communication Data Analyses

7.1 Hypotheses

H3. The banker and the customer share smiles for a longer period of time when the customer

perceives the banker is empathetic.

H4. The banker and the customer share smiles more often when the customer perceives the

banker is empathetic.

As we classified the data into neutral, always smiling and empathetic category according to

three different criteria i.e., cognitively reported customer perception, banker manipulation and

affectively measured facial expressions synchrony in chapter six, we analyze the dyadic

communication with the same way in the following section 7.2, 7.3 and 7.4.

7.2 Analysis 1. Cognitively Reported Customer Perception

We collected forty-six sets of interaction data and in each pair the customer appraised

whether the banker's attitude was neutral, always smiling or empathetic. This section examines when

the data are grouped into these three categories based on using the customer's perceptions of the

banker. Figure 7-1 illustrates the ten kinds of facial expression synchrony between the banker and the

customer that took up the longest duration among 361synchrony pairs in each group. The x-axis

indicates the synchrony ID and the y-axis indicates the fraction of the synchronized facial

expressions. The fraction of each synchrony is computed across the participants in each condition. In

the perceived as "Neutral" condition, the "neutral-caring" pair (ID = 117) occurred most of the time,

taking up 45.1% of the entire interaction time on average. In the perceived as "Always Smiling"

condition, we found more than 50% of one-sided smiles from the banker i.e., 30.8% of the pair

"smile-neutral" (ID = 7), 12.6% of the pair "smile-other" (ID = 8), 11.9% of the pair " smile-

concerned" (ID = 2) and 11.7% of the "smile-surprised" (ID = 10). In these banker-only smiles, the

customer was showing neutral facial expressions rather than smiling, which may be inferred as the

customer not enjoying the interaction. The most frequently occurring synchrony pairs are shown in

Figure 7-2. The x-axis indicates the synchrony ID and the y-axis indicates the number of the

synchronized facial expressions. In the perceived as "Neutral" group, the banker was expressing

neutral facial expressions most of the time, regardless of the customer's facial expressions. And

"neutral-neutral" pair (ID = 121) was the most frequent pair. In "Empathetic", the "neutral-neutral"

pair (ID = 121) occurred 5.6 times on average, which was higher than the value of this pair in the

"Neutral" group, which wa 5. "Smile-smile" pair (ID = 1) was also much more frequent, scoring 6

compared to the other two groups ("Neutral" = 2.8,"Always Smiling" = 3.5). This result implies that

the "Empathetic" group is more dynamic than the other two groups. Therefore, we can conclude that

when the customer perceived the banker as empathetic, they smiled together genuinely more often

and for longer.

Neutral (N=12)60 ID : Banker - Customer

50 117: neutral - caring121: neutral- neutral

40 7 : smile - neutral8 : smile - other

30 118 : neutral - confused122 : neutral - other

20 124: neutral - surprised115: neutral - smile

10 123 : neutral - satisfied2 : smile - concerned

117 121 7 8 118 122 124 115 123 2

Synchronized facial expressions ID

Always Smile (N=19)(%) 60

50

40

30

20

10

0

121 7 116 127 8 2 126 45 10 122


Empathetic (N= 15)(%) 60

50

40

30

20

10

0

121 7 102 8 1 122 10 115 4 119


ID : Banker - Customer121 : neutral - neutral7 : smile- neutral116 : neutral - concerned127 : neutral - bored8 : smile - other2: smile - concerned126 : neutral - interested45 : caring - neutral10 : smile - surprised122: neutral - other

ID : Banker - Customer121: neutral - neutral7 : smile- neutral102 : sorry - neutral8 : smile - other1 : smile - smile122 : neutral- other10 : smile - surprised115: neutral- smile4: smile - confused119: neutral - upset

Figure 7-1. Ten Synchronized Facial Expressions that take up the largest percentage in theentire interaction in each group

Neutral (N=12)7

ID : Banker - Customer6 121: neutral - neutral5 115: neutral - smile

124: neutral - surprised4 1 : smile - smile

117 :neutral -caring3 118 : neutral - confused

2 7 : smile - neutral122: neutral - other

1 4 : smile - confused0 8 : smile- other

121 115 124 1 117 118 7 122 4 8


Always Smile (N=19)7

ID : Banker - Customer6 7 : smile- neutral

5 117: neutral - caring121: neutral - neutral

4 116: neutral - concernedI : smile - smile

3 -115: neutral - smile

2 127 :neutral - bored122 : neutral - other

1 2 : smile- concerned

0 45 : caring- neutral

7 117 121 116 1 115 127 122 2 45


Empathetic (N= 15)7

ID : Banker - Customer6 7 : smile- neutral5 1 : smile - smile

121 : neutral - neutral4 102 : sorry - neutral

115:neutral - smile3 122 : neutral - other

2 8 : smile - other96 : sorry - smile

1 116: neutral - concerned0 2 : smile - concerned

7 1 121 102 115 122 8 96 116 2


Figure 7-2. Ten most frequent synchronized facial expressions in each group

--

7.3 Analysis 2. Banker Manipulation

This section examines when the data are grouped into these three conditions based on using

the banker's manipulation and what he was trying to convey.

Figure 7-3 illustrates the ten kinds of facial expression synchrony between the banker and the

customer that took up the longest duration among 361synchrony pairs in each group. The x-axis



the "Neutral" condition, all of the ten longest synchronies were paired with the banker's neutral facial

expressions. The pair "smile-smile" (ID = 1) happened longer in the "Always Smiling" condition

taking up 16.8% than in "Empathetic" condition, 8.2%. The shared smile pair of "Always Smiling"

condition was observed nearly twice as long as that of "Empathetic" condition. This result can be

accounted for with the facts found in chapter four that explain the difference between what the

banker was trying to do and what the customer was responding with and thinking. In the "Always

Smiling" group, there are more than 50% "smile-neutral" pairs along with other banker-only smile

pairs such as "smile-other"(ID=8), "smile-concerned"(ID=2), "smile-confused"(ID=4), and "smile-

upset"(ID=5). In Figure 7-4, "Empathetic" group shows more diversity in banker's facial

expressions including neutral, smile and sorry. The other two groups do not have "sorry" in banker's

facial expressions, which implies the banker is trying to respond to customer's concerns. Moreover,

the values of the pairs "neutral-neutral"(ID=121), "smile-neutral"(ID=7) and "smile-smile"(ID=1)

seem to be similar between groups; the pair "neutral-neutral"(ID=121) in the "Empathetic" group is

5.2 and in the "Neutral" group is 4.9; the pair "smile-neutral"(ID=7) is 5.4 in the "Always Smiling"

group and 4.9 in the "Empathetic" group; the pair "smile-neutral"(ID=1) is 4.7 in the "Always

Smiling" group and 4.5 in the "Empathetic" group.

Neutral (N=15)(%) 60

50

40

30

20

10

121 116 117 127 118 126 122 124 123 115


Always Smile (N=16)

ID : Banker - Customer121: neutral - neutral116 : neutral - concerned117: neutral - caring127 : neutral - bored118 : neutral - confused126 : neutral - interested122 : neutral - other124: neutral - surprised123 : neutral - satisfied115: neutral- smile

) 60 ID : Banker - Customer

50 7 : smile - neutral10 : smile - surprised

40 8 : smile - other1 : smile - smile

30 2 : smile - concerned4: smile - confused

20 116 : neutral - concerned121 : neutral - neutral

10 115: neutral- smile

5 : smile - upset

7 10 8 1 2 4 116 121 115 5


Empathetic (N=15)(%) 60

50

40

30

20

10

0121 102 122 116 8 115 45

ID : Banker - Customer121 : neutral - neutral102 : sorry - neutral122 : neutral - other116 : neutral - concerned8 : smile - other115 : neutral - smile45 : caring - neutralI : smile - smile7: smile - neutral10: smile- surprised

7 10


Figure 7-3. Ten Synchronized Facial Expressions that take up the largest percentage of theentire interaction in each group

Neutral (N=15)7

6

5

4

3

2

0

121 115 117 127 116 2 122 124 118 119


Always Smile (N= 16)7

6

5

4

3 -

2

07 1 8 121 2 4 5 3 115 116


Empathetic (N= 15)7

6

5

4

3

2

0121 102 7

ID : Banker - Customer121: neutral - neutral115 : neutral - smile117 : neutral - caring127 : neutral - bored116: neutral - concerned2: smile - concerned122 : neutral - other124 : neutral - surprised118 : neutral - confused119: neutral - upset

ID : Banker - Customer7: smile - neutral1 : smile - smile8 : smile - other121 : neutral - neutral2: smile - concerned4: smile - confused5 : smile - upset3 : smile - caring115 : neutral - smile116: neutral - concerned

ID : Banker- Customer121: neutral - neutral102: sorry - neutral7 : smile - neutral1 : smile - smile115 : neutral - smile122 : neutral - other116 : neutral - concerned96 : sorry - smile2 : smile - concerned118 : neutral - confused

1 115 122 116 96 2 118



7.4 Analysis 3. Affectively Measured Synchronized Facial

Expressions

This section examines when the data are grouped into these three conditions based on what

was really observed in their dyadic communication through facial expressions. The classification

rules are described in section 6.4.

Figure 7-5 illustrates the ten kinds of facial expression synchrony between the banker and the

customer that took up the longest duration among 361 synchrony pairs in each group. The x-axis



the "Neutral" condition, except for the "smile-surprised" pair (ID = 10), most of the pairs are

banker's neutral facial expressions. In the "Empathetic" group, the smile pair (ID =1) at 20.2%

appeared the longest compared to the other two groups; there is no shared smile in "the Neutral"

group or in the "Always Smiling" group. In the "Always Smiling" group there is more than 50% of

banker-only smiles, combining 29.0% of "smile-neutral"(ID=7) and 14.0% of "smile-other"(ID=8).

This value is smaller than the total of banker-only smiles in the "Empathetic" group, adding up 27.5%

of "smile-neutral"(ID=7), 18.8% of "smile-other"(ID=8), 13.9% of "smile-surprised"(ID=10), 12.6%

of "smile-concerned"(ID=2) and 8.5% of "smile-confused"(ID=4), yet there is no shared smile that

appears among the ten longest pairs in "Always Smiling". The pair "smile-smile"(ID = 1) is most

frequent in the "Empathetic" group compared to other two groups (Figure 7-6).

Neutral (N=15)(%) 60

50

40

30

20

0

121 116 127 117 122 118 126 10 123 115


Always Smile (N=15)

ID : Banker - Customer121: neutral - neutral116 : neutral - concerned127 : neutral - bored117: neutral - caring122 : neutral - other118: neutral - confused126 : neutral - interested10 : smile - surprised123 : neutral - satisfied115: neutral- smile

(%) 6o ID : Banker -Customer

50 121: neutral - neutral7: smile - neutral

40 102: sorry - neutral8 : smile - other

30 122: neutral - other116: neutral - concerned

20 124: neutral- surprisedI0 115: neutral -smile

10 118: neutral- confused

0 45: caring -neutral

121 7 102 8 122 116 124 115 118 45


Empathetic (N= 16)(%) 60 ID : Banker - Customer

50 7 : smile- neutral45 : caring - neutral

40 1 : smile - smile8 : smile - other

30 121: neutral - neutral10 : smile - surprised

20 2: smile - concerned115 : neutral - smile

10 4: smile- confused122 : neutral - other

7 45 1 8 121 10 2 115 4 122


Figure 7-5. Ten Synchronized Facial Expressions that take up largest percentage in the entireinteraction in each group

slr~

Neutral (N= 15)7

ID : Banker - Customer6 121: neutral- neutral

116: neutral - concerned115 : neutral - smile

4 117: neutral-caring127 : neutral - bored

3 122: neutral- other

2 2: smile-concerned7 : smile - neutral

1 118: smile - confused1 : smile - smile

0121 116 115 117 127 122 2 7 118 1


Always Smile (N= 15)7

6

5

4

3

2

0121 102 115 7 116 96 122 1 118 8

ID : Banker - Customer121: neutral - neutral102 :sorry - neutral115: neutral - smile7 : smile - neutral116 : neutral - concerned96 : sorry - smile122 : neutral - other1 : smile - smile118 : neutral - confused8 : smile - other


Empathetic (N= 16)7

ID : Banker - Customer6 7: smile - neutral

5 1 : smile - smile45 : caring - neutral

4 121: neutral- neutral115 : neutral - smile

3 122 : neutral - other

2 8: smile - other2 : smile - concerned

1 3: smile - caring4: smile - confused

07 1 45 121 115 122 8 2 3 4



1

7.5 Summary

We measured the percent of each facial expression synchrony pair as well as the number of

each facial expression synchrony pair. Both measures of the smile-smile pair appeared highest when

the customer perceived the banker was empathetic, which confirms the hypotheses three and four.

Note that the shared smile pairs were more distinctively found when the data were classified according

to what actually happened in the interaction compared to the analyses done by customer perception

and banker manipulation. In the next chapter, chapter eight, we suggest a more novel way to

understand smiles by their context and intent and classify them into four kinds of smiles.

Chapter 8

Specific Classification and Analysis of

Smiles

8.1 Smiles with Context

As is explained in section 5.1, the data gathered for each participant includes the time, facial

expressions and a short comment about the facial expressions. From these comments we can infer

whether the person was smiling just for greeting, laughing at a joke, satisfied with the information or

trying to be polite. Therefore we classified the smile labeling into three groups; social, genuine, and

greeting smile. Here are some examples of the comments that can be classified into these three kinds

of smile (Figure 6-4). Interestingly, there are positive reasons for genuine smiles but also negative

reasons for genuine smiles such that the customer smiled when he disagreed with the banker,

probably politely, or when he encountered an unexpected situation where did not get the full amount

of compensation for the study, probably to make the banker feel it is okay.

Figure 8-1. The examples of social, genuine, and greeting smiles

8.2 Validation of Social, Genuine and Greeting Smiles

Three undergraduate students labeled three kinds of smiles as explained in section 8.1. Three of them

sat down together until they were trained to get agreement between these three kinds of examples and

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;~

then they were separated and labeled the smiles of the full set of data. There are forty-six interactions

and four labeled files for each interaction. There are 46*4 = 184 labeled files in total and 60 of them

did not have any smiles labeled in them. Therefore there were 124 smile labeled files and three

coders labeled these files with social, genuine and greeting smiles. Figure 8-2 shows the inter-coder

agreement using Cohen's Kappa coefficient. Cohen's Kappa coefficient is a commonly used measure

in social science analysis to take into account the effect that human coders can show agreement by

chance when there are two coders (Cohen, 1960; Fleiss, 1971; Galton, 1892; Gwet, 2001; Landis,

1977). From this result, 67 files among 124 smile labeling files got 0.81-1.00 of Cohen's Kappa

coefficient, which means the three coders labeling was pretty much in agreement.

80

70

60

- 50 -t9

40 K Interpretation<0 No agreement

-30 - 0.0-0.20 Slight agreement0.21-0.40 Fair agreement

0.61-0.80 Substantial agreement0.81-1.00 Almost perfect agreement

i2 U

0-

<0.0 0.0-0.20 0.21-0.40 0.41-0.60 0.61-0.80 0.81-1.00

Cohen's Kappa Coefficient (K)

Figure 8-2. Cohen's Kappa Coefficient among three coders

.i. .. .. ...! .... ... ... ..... ...... .... .......................................... ......

I1

Chapter 9

Conclusions and Future Work

9.1 Summary

"Service with a smile" (Pugh, 2001 & Grandey, 2005) is a common motto in the service

provider industries. It is assumed that a smiling employee is the best representative for customer

interaction. We examined the effects that a service provider's facial expression manipulation during

customer interaction had on customer satisfaction. We measured interaction satisfaction, information

satisfaction, and the perceived empathetic attitude of the service provider, to estimate customer

satisfaction. The results show that customer satisfaction was not significantly affected by the

banker's intent to portray a particular quality of interaction. Rather, customer satisfaction was

dependent on how the customer perceived the service provider's attitude. When the customer

perceived the service provider as empathetic, customer satisfaction was greater than the cases in

which the customer thought the service provider was neutral or always smiling. We looked more

specifically at the data when the customer felt the service provider was empathetic, and measured

that they shared smiles together more often and for a longer time. In contrast to the common notion

of "Service with a Smile", the "Always Smiling" attitude of the service provider was not effective in

making the service provider or experience more enjoyable. Based on measuring smiles, the important

aspect of smiles in service appears to be when conditions are such that both the service provider and

the customer smile, together, and genuinely. This principle also applies to other complementary

facial expressions.

9.2 Suggested Future Work

Dyadic analysis of forty-six face-to-face customer service encounters reveals that affective

social signals through facial expressions have a significant influence on how customers perceive the

service. We obtained this result using the facial expression labels provided by the experimental

participants. However, the labeling depends on the labeler's perspective. To improve the reliability of

the data labeling, we could compare the three different labelings done by the participants themselves,

their interaction partners and four independent outside coders. In addition, we classified the smiles

into three kinds of smiles; specifically it becomes four kinds when the genuine smile is divided into

positive and negative ones, in chapter eight and we suggest to analyze customer satisfaction and

dyadic communication with these labeled data.

The current state of computer vision technology suggests that computers can be trained to

recognize a number of human facial expressions automatically (el Kaliouby, 2005 & Zaman, 2006).

We visualized the dyadic pattern of facial expressions between a service provider and a customer by

assigning each facial expression to a specific color. A future application might provide real-time

feedback (e.g., Kim et al., 2008) about affective information during a service interaction. With regard

to data analysis, other measures such as the duration of each facial expression and the number of

transitions of between expressions, may be useful. In addition, for more complete understanding,

voice analysis can be combined with facial analysis to explain such topics as why a customer

perceives a service provider was smiling even when the service provider intended to be neutral.

While sharing genuine smiles appears to be very important to interaction, the way to best achieve this

sharing is more challenging than simply asking the banker to act empathetically.

Appendix A: 361 Banker and Customer

Facial Expressions Synchrony IDs

uCstomet Missingsmile concemed caring confused upset sorry neutral other Satisfied Surprised Relieved Interested Bored Enthusiatic ersuasiv Annoyed Survey The end

ankeLabel

Smile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Concerned 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

Caring 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57

Confused 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76

Upset 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95

Sorry 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114

Neutral 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133

Other 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152

Satisfied 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171

Surprised 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190

Relieved 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209

Interested 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228

Bored 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247

Enthusiastic 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266

Persuasive 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285

Annoyed 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304

Survey 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323

The end 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 349 340 341 342

Missing343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361

Label

Bibliography

[1] Alford, Bruce L. and Daniel Sherrell. (1996). The Role of Affect in Consumer Satisfaction

Judgments of Credence-Based Services. Journal of Business Research, Vol. 37, 71-84.

[2] Baaren, R. (2003). Mimicry for money : Behavioral consequences of imitation. Journal of

Exprimental Social Psychology, Vol. 39, Issue 4, 393-398

[3] Bagozzi, Richard P., Mahesh Gopinath, and Prashanth U. Nyer. (1999). The Role of Emotions in

Marketing. Journal of the Academy of Marketing Science, Vol. 27, No. 2, 184-206.

[4] Barger, P. (2006). Service With a Smile and Encounter Satisfaction: Emotional Contagion and

Appraisal Mechanisms. Academy of Management Journal, Vol. 49, No. 6, 1229-1238

[5] Barsade, Sigal G. (2002). "The Ripple Effect: Emotional Contagion and its Influence on Group

Behavior." Administrative Science Quarterly, 47, 644-675

[6] Chartrand, T.L. & Bargh, J.A. (1999). The chameleon effect: The perception-behavior link and

social interaction. Journal of Personality and Social Psychology, Vol. 76, 893-910.

[7] Chartrand, T.L., Maddux, W, & Lakin, J. (2005). Beyond the perception-behavior link: The

ubiquitous utility and motivational moderators of nonconscious mimicry. (R. Hassin, J. Uleman, &

J.A. Bargh, Eds.), The New Unconscious (pp. 334-361). New York: Oxford University Press.

[8] Cohen, Jacob (1960). A coefficient of agreement for nominal scales, Educational and

Psychological Measurement Vol.20, No. 1, 37-46.

[9] Decety, J. (2004). The Functional Architecture of Human Empathy. Behavioral and Cognitive

Neuroscience Reviews, Vol. 3, No. 2, 71-100

[ 10] Decety, J. (2007). A social cognitive neuroscience model of human empathy. (E. Harmon-Jones

& P. Winkielman, Eds.), Social Neuroscience: Integrating Biological and Psychological Explanations

of Social Behavior (pp. 246-270). New York: Guilford Publications.

[ 11] Decety, J. & Sommerville, J.A. (2003). Shared representations between self and others: A social

cognitive neuroscience view. Trends in Cognitive Sciences, Vol. 7, 527-533.

[12] el Kaliouby, R., Robinson, P. (2005). Real-time inference of complex mental states from facial

expressions and head gestures. Real-time vision for HCI, 181-200

[13] Fleiss, J.L. (1971). "Measuring nominal scale agreement among many raters." Psychological

Bulletin, Vol. 76, No. 5, 378-382

[ 14] Fleiss, J. L. (1981) Statistical methods for rates and proportions. 2nd ed. (New York: John

Wiley); 38-46

[15] Fleiss, J.L. and Cohen, J. (1973) "The equivalence of weighted kappa and the intraclass

correlation coefficient as measures of reliability" in Educational and Psychological Measurement,

Vol. 33, 613-619

[16] Galton, F. (1892). Finger Prints Macmillan, London.

[17] Grandey, A.A., Fisk, G.M., Mattila, A.S., Jansen, K.J., Sideman, L.A. (2005), "Is 'service with a

smile' enough? Authenticity of positive displays during service encounters", Organizational

Behavior and Human Decision Processes, Vol. 96 No. 1, pp. 3 8 -5 5 .

[18] Gwet, K. (2001) Statistical Tables for Inter-Rater Agreement. (Gaithersburg : StatAxis

Publishing)

[19] Hatfield, E., Cacioppo, J. T., & Rapson, R. L. (1994). Emotional contagion. Cambridge:

Cambridge University Press.

[20] Hochschild, A. R. (1983). The managed heart. Los Angeles: University of California Press.

88

[21] Kendon, A. (1970). Movement coordinatioon in social interactions. Acta Psychologica, Vol. 32,

101-125.

[22] Kenny, D.A., Kashy, D.A., and Cook, W.L. (2007). Dyadic Data Analysis, Vol. 61, 263-295,

Guilford Pubn

[23] Kim, T. Chang, A. & Pentland, A. (2008). Meeting Mediator: Enhancing Group Collaboration

with Sociometric Feedback, In Proceedings of Conference on Computer Supported Collaborative

Work, 457-466

[24] Knoblich, G. & Flach, R. (2001). Predicting the effects of actions: interactions of perception and

action. Psychological Science, Vol. 12, 467-472.

[25] Knoblich, G., & Flach, R. (2003). Action identity: Evidence from selfrecognition,prediction, and

coordination. Consciousness and Cognition, 12, 620-632.

[26] LaFrance, M. (1982). Posture mirroring and rapport. (M. Davis, Ed.). Interaction rhythms:

Periodicity in communicative behavior (pp. 279-298). New York: Human Sciences Press.

[27] Lakin, J. L., Jefferis, V. E., Cheng, C. M., & Chartrand, T. L. (2003). The Chameleon Effect as

social glue: Evidence for the evolutionary significance of nonconscious mimicry. Journal of

Nonverbal Behavior, Vol. 27, 145-162.

[28] Landis, J.R. and Koch, G. G. (1977) "The measurement of observer agreement for categorical

data" in Biometrics. Vol. 33, 159-174

[29] Lipps, T. (1903). Grundlegung der Asthetik. Hamburg/Leipzig, Germany:Leopold Voss.

[30] Maurer, R. E., & Tindall, J. H. (1983). Effect of postural congruence on client's perception of

counselor empathy. Journal of Counseling Psychology, Vol. 30, 158-163.

[31] Oliva, Terence A., Richard L. Oliver, and Ian C. MacMillan (1992), A Catastrophe Model for

Developing Service Satisfaction Strategies, Journal of Marketing, 56 (July), 83-95.

[32] Oliver, Richard L. (1980), A Cognitive Model of the Antecedents and Consequences of

Satisfaction Decisions, Journal of Marketing Research, 42 (November), 460-69.

89

[33] Oliver, R. L. (1997). Satisfaction: A Behavioral Perspective on the Consumer. New York: The

McGraw Hill.

[34] Parasuraman, Zeithaml & Berry (1985). "A Conceptual Model of Service Quality and Its

Implications for Future Research," Journal of Marketing, 41-50.

[35] Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). SERVQUAL: A multiple-Item scale

for measuring consumer perceptions of service quality. Journal of Retailing, Vol. 4, 12-39.

[36] Prinz,W. (1997). Perception and action planning. European Journal of Cognitive Psychology, 9,

129-154.

[37] Pugh, S. D. (2001). Service with a smile: Emotional contagion in the service encounter.

Academy of Management Journal, Vol. 44, 1018-1027

[38] Rafaeli, A. (1989). When clerks meet customers: A test of variables related to emotional

expression on the job. Journal ofApplied Psychology, Vol. 74, 385-393.

[39] Rafaeli, A., & Sutton, R. I. (1989). The expression of emotion in organizational life. In L. L.

Cummings and B. M. Staw (Eds.), Research in organizational behavior, vol. 11: 1-42. Academy of

Management Journal - In Press 24 Greenwich, CT: JAI Press.

[40] Rafaeli, A., & Sutton, R. I. (1990). Busy stores and demanding customers: How do they affect

the display of positive emotion? Academy of Management Journal, Vol. 33, 623-637.

[41] Tsai, W. (2001). Determinants and consequences of employee displayed positive emotions,

Journal of Management, Vol. 27, 497-512

[42] Tsai, W.-C., & Huang, Y.-M. (2002). Mechanisms linking employee affective delivery and

customer behavioral intentions. Journal of Applied Psychology, Vol. 87, 1001-1008.

[43] Wilk, S.L. & Moynihan, L.M. (2005). Display rule "regulators": The relationship between

supervisors and worker emotional exhaustion. Journal ofApplied Psychology, Vol. 90, 918- 927.

[44] Zajonc, R. B. (1985). Emotion and facial efference: An ignored theory reclaimed. Science, 5:

15-21.

[45] Zaman, B. (2006). "The FaceReader: Measuring instant fun of use," NordiCHI2006

affect reflection technology in face-to-face service

Documents