Emotional condition in the Health Smart Homesenvironment: emotion recognition using ensemble

of classifiersLeandro Y. Mano

Institute of Mathematical and Computer SciencesUniversity of Sao Paulo

Sao Carlos, Brazilleandroyukiomano@usp.br

Abstract—The fraction of elderly people in the populationhas increased significantly in most developed countries in recentyears. Embedded computing enables the deployment of HealthSmart Homes that can enhance nurses or caregivers to assistpatients in need of timely help, and the implementation of thisapplication can be extremely easy and cheap when aided byInternet of Things technologies. In addition, there are few studiesthat take into account the patient’s emotional state, which iscrucial for them to be able to recover from a disease. Thisarticle discusses the use of images and emotional detection toassist patients and elderly people within an in-home healthcarecontext and sets out a model for the classification of emotionsbased on the facial expressions of the users. Finally, the resultsdescribed in this article show that the Ensemble Classificationthat is put forward can achieve greater rates of accuracy, averageof 82,53%, in classifying feelings than what can be obtained byusing a single classifier.

Keywords-Internet of Things; Health Smart Homes; EmotionsClassification; Facial Mapping; Human-Computer Interaction.

I. INTRODUCTION

The number of elderly people is large increasingly in severalcountries, highlighting countries in Europe, the USA, Japanand Brazil. These people tend to live alone and are prone tohave more diseases than young people. In view of this, the useof Health Smart Homes (HSH) is extremely important [1],[2]. The concept of HSH has emerged from a combinationof telemedicine, domotics products and information systems;it can be defined as a Smart Home that is equipped withspecialized devices for remote healthcare. These devices aremainly sensors and actuators that can take action whenever acritical situation is detected. The aim is to allow the technologyto be used to monitor patients and elderly people and issuealerts to nurses and/or relatives whenever necessary whilethey are recovering at home, in addition to providing someindependence so that they can live their lives in a more self-reliant manner.

Apparently, most current and recent research focuses onimproving the daily lives of patients by providing them with

Second affiliation: Johann Bernoulli Institute for Mathematics and Com-puter Science - University of Groningen, Groningen, The Netherlands.

Foundation for Research Support of the State of Sao Paulo - FAPESP(process ID 2016/14267-7 and 2017/21054-2).

The copyright: 978-1-5386-5150-6/18/$31.00 2018 IEEE.

specific gadgets, such as alert alarms; however, it cannotrecognize any type of emotional response which is a crucialfactor during the treatment at home to determine the emotionalhealth of an individual in turn, can be used as an importantsymptom for the diagnosis of various diseases [3]–[5] suchas: schizophrenia, depression, autism and bipolar disorder.We highlight that the above mentioned diseases may besymptoms caused by the excessive time on negative emotions,lack of emotional expressions or the instability of expressedemotions [3]–[5]. In addition to the use of emotional healthto diagnose diseases, emotional states can influence socialinteraction and also help us in making decisions/measures ifhe/she is stressed.

This paper offers an in-home healthcare solution by makinguse of images and facial expressions to monitor patients andelderly people that have special needs. Hence, this articleproposes a solution to intelligently detects user emotions.In our approach, the emotional classification is carried outthrough the use of Machine Learning (ML) techniques, sothat emotions can be detected and classified. Capturing facialexpressions over a certain period of time can give an idea of towhat extent the elderly/patient is feeling pain and can enablea nurse/relative to decide whether help is required or not.Similarly, the patients face can register his/her emotional state,which is also a crucial factor during the treatment. Emotionplays an important role in recovery from disease [6], [7]. Thepurpose of our approach is to detect any new feature in thehouse through the use of cameras and facial expressions. Itshould also be stressed that there is no need to have someonewatching the whole video stream all day long as the detectionof new features is carried out automatically without any humanintervention.

While having a camera in the home raises problems aboutprivacy, we believe these can be overcome since they can beturned off whenever a resident wishes. It should be noted thatthe use of cameras and image processing can be less intrusiveas there is no need for people to be fitted with any special item(such as wireless-based emergency buttons) to ask for specialhelp.

This article is structured as follows. Section II, there is areview of the literature with regard to the issues discussed

in this article and how these studies can be drawn on toform an emotion recognition and classification system in HSH.Section III examines some important concepts that are neededto understand the use of the face to identify emotions, wedescribe how the face mapping is performed, we set out theemotion classification module and our approach for monitoringemotional aspect in HSH. In Section IV, there is an assessmentof the emotion classification system that was conducted tovalidate our approach. Finally, Section V concludes this workand some suggestions are made for future investigations.

II. RELATED WORK FOR HEALTH CARE CONSIDERINGEMOTIONS

This section examines the advances made in HSH based onInternet of Things (IoT) technologies with a view to describingand explaining the main challenges of this research study.The authors of [8] review the emerging concept of Health“Smart” Home and picture potential applications of thesesystems, such as social alarm systems and remote monitoring.The paper also discusses prominent technologies that arebeing deployed and reports pilot projects of e-care systemsalready in use by hospitals and residences. The advances ofin-home healthcare are also due to the application of AmbientIntelligence and Artificial Intelligence. The combination ofmodern sensing equipments, with advanced data processingtechniques and wireless networks culminates in the creation ofdigital environments that improves the daily life of residents.

The study of [9] used cameras for improving the analysis ofthe environment, e.g. detecting possible hazardous situations,while providing a system that is non-intrusive. Different fromthis work that argues that the use of a camera-based approachenables the emotional state of the patients to be analyzedand can provide a non-intrusive system that is not heavilydependent on attached sensors, unlike most of the currentsolutions. The proposed solution differs from the other systemsand introduces new features for treating patients.

The authors of [1] propose a wireless architecture forPersonal Area Network (WPAN) that takes advantage ofthe benefits offered by an intelligent environment by usinginformation from sensors. The system is based on imageprocessing and can provide solutions that are integrated withother control devices. This enables the detection of movementsand patterns of the activities of a resident, such as speed anddirection. It should be noted, however, that no attempt is madeto analyze the emotional state of the residents in order to assessthe state of their health.

Another important work is being developed by the Univer-sity of Florida’s Mobile and Pervasive Computing Laboratory,creating the Gator Tech Smart House (GTSH) [10], an exper-imental laboratory and a real environment aimed at validatingsmart home technologies and systems. The goal of GTSH isto create supportive environments, such as homes, that canhelp its residents and is able of mapping the physical worldwith remote monitoring and intervention services, conductingresearch and development activities designed to assist elderlyand people with special needs in maximizing independence

and maintaining a high quality of life. Similarly, researchesfrom the Georgia Institute of Technology developed the AwareHome Research Initiative (AHRI) [11]. The Aware Home hasenabled research in many areas, such as: provision of data tothe house, innovative sensing and control infrastructure, anda variety of applications to assist residents. The health andwell-being of residents is one area of research that has receivedsignificant efforts focusing on technologies that enable a betteraging, and help caregivers of children with disabilities suchas autism. The Aware Home is a residential laboratory andsuch studies have achieved satisfactory results enabling realresidents to use the developed prototypes. However, in none ofthe two approaches, an attempt is made analyze the emotionalstate of the residents in order to assess the state of their health.

Table I summarizes the related literature is made providingthe following comparison items to ensure that an overallview is given: (i) use of IoT sensors while monitoring pa-tients/elderly people; (ii) adoption of image processing tech-niques as a way of monitoring people, and; (iii) employmentof user’s emotions for identifying patient’s well-being.

TABLE ISUMMARY OF THE PAPERS IDENTIFIED IN THE LITERATURE.

Related work Use of IoT Image Considerreferences sensors processing emotions

Romero et al., 2009 [1]√

X XAugusto et al., 2007 [9] X

√X

Helal et al., 2005 [10]√

X XKientz et al., 2008 [11]

√X X

Our system√ √ √

To the best of our knowledge, none of the proposals inthe literature take into account the patient’s emotional sideand hence they fail to exploit a considerable amount of “rich”information available in facial expressions and body language.This includes detecting pains and other emotional relatedissues, as depression and autism.

III. OUR SMART IN-HOME HEALTHCARE SYSTEMCONSIDERING EMOTIONS

This work adopts a generic approach to construct adaptiveapplications on computing devices, making use of facialexpressions to monitor patients and elderly people that havespecial needs. Thus, this section we describe how the facemapping is performed by the image, we set out the emotionclassification module based on ML algorithms and our ap-proach for monitoring emotional aspect in the environmentHSH.

A. Study of Emotions and Facial Mapping

Motor Expressions which are also known as ExpressiveReactions are responsible for communicating the behavioraltendencies of an individual [12]–[15]. This emotional featureinvolves the changes in facial expression that register theemotional experience of the users.

Basic emotions are represented by distinct and universalfacial expressions. The word universal suggests that the same

muscular movements are involved in carrying out facial ex-pressions. Within the discrete models, the most widespreadis [16], which states that there are the following basic emo-tions: joy, disgust, fear, a neutral state, anger, surprise andsadness - as can be seen in Fig. 1. All the other emotionalcategories can then be built on combinations of these basicemotions. The main advantage of using this model is thatpeople tend to describe emotional demonstrations observed intheir day-to-day life very similarly. Thus, these models facil-itate the association of emotions with the facial expressionsthat represent them.

Fig. 1. Basic emotions proposed by [16], photos used by members of theresearch group.

As shown by [17], the task of analyzing automatic facialexpressions can be divided into three key stages: i) facedetection, ii) facial feature extraction and iii) emotion clas-sification. Face detection is a processing stage to automati-cally find the facial region on the input images. Detectinga face in a complex scene is a non-trivial problem: headmovements, occlusion, changing the illumination settings, andthe presence of hair or glasses, are examples of possiblecomplications that can occur to the system [17]. After theface detection, it is possible to extract and represent the facialchanges produced by expressions, because several types ofperceptual cues to emotional states, are displayed in the face.The extraction approach for face identification and expressionanalysis adopted here, is based on geometric features. Methodsbased on geometric features are used in facial modelling(motor expressions) with a view to adopting an approachwhich resembles the way that human beings interpret thedifferent parts of the face. Thus, the use of different facial rep-resentations (a neutral state, joy, disgust, sadness, fear, angerand surprise) is a recommended way for the identification andclassification of feelings, since they are able to encode thefacial configuration of the individual [12], [16], [18]. Thus, it ispossible to model the shape and locations of facial components(including the mouth, eyes, eyebrows and nose) through theuse of geometric elements such as angles, distances or areasused to represent the geometry of the face. The classification

of facial expressions is the last stage where techniques basedon ML are employed to tackle this problem.

In this paper, FaceTracker [19] is used for facial map-ping, being a tool of computer vision that is used to obtaininformation about facial features. It uses an optimizationstrategy which employs a linear approximation by adjustingthe reference points in consistent locations to record a modeldesigned to work within fixed parameters; moreover, the algo-rithm seeks to align the elements of the face being analyzedwith the feature points of the reference model. To reducethe computational cost while at the same time being able toclassify emotions, we considered thirty-three important facialpoints that are crucial to this end [20]. Let us mention thatthis is supported by the theories of the psychologists [16]).As such, we considered i) eight points that map the mouth,ii) six for each individual eye, iii) three for each eyebrow; iv)three for the chin, v) two for the nostrils. We also consideredvi) two for delineating the lateral extremities of the face nearthe eyes. Figure 2 provides an example of the mapping of aface carried out by FaceTracker that was used in our model.

Face mappedparameters

(1.130 features)

Face mapping process

33 importantfacial points

All possiblecombinations of

the points(distances and

angles)

Fig. 2. Face mapping process by FaceTracker [20], photos used by membersof the research group.

To compute the number of features, we considered thedistances between two distinct points and the angles madeby the line connecting them with the horizontal axis. Theseare obtained at all possible combinations of the points. Asa result, it creates a dimensionality representation of 1.130features. These features are used for the training of our modelfor emotion classification.

B. Approach to classification of emotionsDespite their extensive use, the classifiers generated by

means of ML methods rarely achieve a 100% degree ofaccuracy [21]. This is because the performance of each methoddepends on several parameter settings, how representative arethe data from the training set and the degree of difficultyassociated with particular problem. Selecting a single classifierinvolves rejecting a significant amount of potentially usefulinformation. For this reason, the concept of Ensemble ofClassifiers (EC) has been regarded as a possible solution forthe development of high-performance systems in the area ofpattern recognition [22]. This is based on the assumption thatthe combination of classifiers can lead to an improved perfor-mance in terms of generalization and/or increased accuracy.

The classifiers that compose an EC are assumed to bedifferent from each other if they have uncorrelated errors.Thus, for an ensemble to achieve an acceptable performance,it must avoid making coincidence errors, so that the errors ofa classifier can be corrected by the choices made by all theother components [21], [22]. For this reason, for our proposalclassification techniques were used that have been increasinglyemployed to analyze the user’s emotional responses, such as:

• k-Nearest Neighbor (kNN) - The kNN, used in some stud-ies [23], [24] is a non-parametric classification approach,which assumes no prior probability distribution for thedata. The kNN algorithm is a set of k objects in thetraining set that are closer to the test object, and can beclassified as the predominant class of its k-neighbors [25].

• Support Vector Machine (SVM) - The SVM algorithmused by [26]–[28] for emotion recognition is one ofthe most robust and accurate methods of the knownalgorithms. According to [25], the goal of SVM is to findthe best function to distinguish two classes of trainingdata. It ensures that the best function is achieved becauseit maximizes the margin between the classes by offeringa better generalization ability [27], [28].

• Logic Fuzzy - Logic Fuzzy, which is adopted and usedin [26], [29], is capable of processing incomplete data andproviding approximate solutions to problems that othermethods are not able to solve [29].

• Decision or Regression Tree - The Decision or RegressionTree applied by [23], [26] uses a tree to perform theclassification or value estimation of a given test. Oneof the most well-known algorithms that generates thesetrees is C4.5. This algorithm grows the tree through adivide-conquer approach; the tree is branched by usingthe attribute that obtains the best information gain [30].

• Bayesian Networks - Bayesian Network is a probabilisticgraph model, which was applied by [26], [27], [31] andrepresents a set of random variables and their conditionaldependencies as a directed acyclic graph [25]. For exam-ple, a Bayesian network may represent the relationshipbetween symptoms and diseases. Once the the symptomsare known, one can infer the probability of the outbreakof a disease.

On the basis of this assumption, Figure 3 shows the modulestructure for face which is classified by means of the ECapproach. This model aims to use a facial mapping and,through a combination of response values of classificationalgorithms, identify and categorize the user’s emotion, sothat computing systems can interact more assertively with theuser’s emotional state.

The first processing layer (layer 1 of Figure 3) receives, asinput, the results of the facial mapping. This layer consistsof individual classifiers that may have different architectures,but outputs a common form, such as the ranking of candidateclasses. The second processing layer (layer 2 of Figure 3)consists of a decision-making process which operates on theoutputs of the previous layer to generate the general decision

Fig. 3. Architecture of our approach: Ensemble of Classifiers to detectemotion through the user’s face.

of the EC (see Figure 3). Equation 1 was used for theweighting process and assigned proportional weights for eachalgorithm in accordance with its rate of accuracy [20]. In theequation, W is the weighted average of accuracy for eachalgorithm; AR matches the average rate of accuracy for thealgorithm. Thus, each algorithm will be assigned a weight thatis proportional to its degree of accuracy.

Wxi =ARxi∑nj ARxj

(1)

C. Generic Approach for Emotional Aspect

With this in mind, a system called FlexEmotion was de-signed that is based on the OpenCom model [32], which hasthe following characteristics: i) a generic software constructingsystem; ii) flexible and extensible architecture; iii) it is alanguage-independent plataform; iv) it is based on a micro-kernel, where the features are deployed whenever required.

FlexEmotion is a framework that supports the developmentof systems that are able to classify to the emotional state of theusers during the interaction [32]. Figure 4 illustrates the FlexE-motion architecture, in which the microkernel entity loads andinterconnects the components needed for the interaction withthe user. In the case of this work, FlexEmotion instantiates andconnects a camera (on the right of Figure 4), but it can alsobe used for other sensors, e.g. video, temperature or sound.

Camera imageSensors

Sound

temperature

FlexEmotion

Fig. 4. Relationship between components on FlexEmotion, based on themodel OpenCom [32].

We propose a system based on the use of IoT for smart andindividualized monitoring of elderly patients in their homes.Having as main features the scalability of its componentsand the ability to infer the emotions or feelings that patientsexperience (such as pain or the onset of depression).

The system is based on the Smart Architecture for In-Home Healthcare (SAHHc) [18], composed of two elements:(i) Sensors; and (ii) the Decision Maker. The sensor elementsare devices distributed in the environment; they can be foundin large numbers and their purpose is to collect information(for instance, capture images) on the patient’s health and sendthis to the Decision Maker element. Furthermore, the DecisionMaker serves as an interface between the patient and thehelper. Figure 5 shows the elements in a scenario where theproposed system is used.

Decision Maker

Sensors

camera

camera

camera

camera

Fig. 5. Architecture proposal for interaction with the computing device [18].

Given the features of the environment where this type ofapplication has to be implemented, all the devices must runon the power supply of the household and use battery as abackup source. In addition, it is assumed that the residence hasan internal communication network that can be used by thesystem. Although wireless networks ensure that the system canbe implemented with ease and mobility, a wired network canalso be used. The structure consists of simple and dedicateddevices that act as sensors and collect the information aboutthe patient’s emotion. The sensor elements are able to detectthe presence of people and also capture their images andthe information is processed to classify the person’s facialcharacteristics and detect his/her emotional state. Figure 6depicts the structure of the interactions between FlexEmotionand the user. This framework includes a recognition module,which extracts the user’s facial features, and the classificationalgorithm, that classifies an emotion based on the featuresprocessed in the identification of the user’s face.

IV. EVALUATION AND DISCUSSION

For training/evaluating our model, we used pictures offacial expressions with different emotions to assess the En-semble of Classifiers; the databases used are Radboud Faces(RaFD) [33], Extended Cohn-Kanade (CK+) [34], IMPA-FACE3D [35] and FACES [36], all with open access tothe public. The RaFD database provides examples of thefacial expressions of 67 participants (adults and children, maleand female), of American and Moroccan nationality. CK+

Health Smart Homes

Facial mapping

Emotion interaction

Fig. 6. System proposal for interaction with the user.

database only has facial expressions of adults, 69% female and31% male, 81% are Europeans or Americans, 13% African-Americans and 6% belong to other ethnic groups. CK+ has593 facial expressions of 123 different participants. IMPA-FACE3D database includes pictures of 38 individuals, withsamples of the six universal facial expressions and neutralexpression. It consists of 22 men and 16 women, most agedbetween 20 and 50. Finally, the FACES database consists ofimages of the faces of 171 subjects, 58 young, 56 middle-agedand 56 elderly men and women, showing each of the sevenfacial expressions analyzed in this work.

Only the subsets of these databases are used in the exper-iments. We selected all the images in which the participantswere facing forward and expressing the following emotions:joy, disgust, fear, anger, surprise, sadness or in a neutralstate. Thus, 67 examples of each emotion are obtained fromthe RaFD database, 38 from IMPA-FACE3D and 171 fromFACES, including children, adults and elderly people. Theselected subset from the CK+ database, includes all theexisting expressions from the different participants, for eachof the evaluated facial expressions. Thus, we used a total of3.115 images in our experiment, divided into 445 images foreach emotion group, to achieve an optimal balance betweenthe training images used for the tests. It is worth notingthat the combination of the selected images, provided a setof heterogeneous tests, consisting of images from differentregions and a model that can be applied to any computingdevice regardless of the ethnicity of the individual who usesit.

Our approach used the WEKA Framework [37] to facilitatethe implementation of the selected algorithms. This is widelyemployed by researchers in the area of ML and Data Mining.This framework is also suitable for the development of newML systems [37]. Table II shows the parameters used for theindividual tests as well as for the development of the proposedEnsemble model.

In this context, we implemented the algorithms to assess theaccuracy of the classification module. As mentioned earlier,the approach were adopted to set the weights that make up theEC: weighted average. As a result, we were able to collect dataon the behavior and evaluation of the classification module as

TABLE IIPARAMETERS USED IN THE SELECTED CLASSIFIERS OF WEKA.

Implementation ParametersdisplayModelInOldFormat = false,

Bayesian useKernelEstimator = false,(NaiveBayes) useSupervisedDiscretization = false

binarySplits = false, collapseTree = true,confidenceFactor = 0.25, debug = false,

Decision Tree minNumObj = 2, reducedErrorPruning = false,(J48) numFolds = 3, saveInstanceData = false,

seed = 1, subtreeRaising = true, unpruned = false,useLaplace = false, useMDLcorrection = true

KNN = 10, debug = false, fuzzifier = 3.0,Fuzzy similarity = SimilarityFNN -R first-last -T

(FuzzyNN) weka.fuzzy.tnorm.TNormLukasiewicz -C 0.0KNN = 1, crossValidate = false, debug = false,

distanceWeighting = No distance weighting,kNN nearestNeighbourSearchAlgorithm =(IBk) LinearNNSearch -A

“weka.core.EuclideanDistance -R first-last”,meanSquared = false, windowSize = 0buildLogisticModels = false, c = 1.0,

checksTurnedOff = false, debug = false,SVM filterType = Normalize training data,

(SMO) kernel = PolyKernek -C 250007 -E 1.0,numFolds = -1, randomSeed = 1,

epsilon = 1.0E-12, toleranceParameter = 0.001

a whole. The model takes as input the user’s facial featuresfor the automated emotion classification.

A. Evaluation of the Emotion Classification

Initially, we analyzed the performance of the algorithmsseparately through the Experimental Planning and Evaluationtechnique; in this case, we employed k-fold cross-validationwith k = 10, with k - 1 for training and the rest for test-ing. Thus, it is possible to measure error estimation moreaccurately, as the mean value estimate tends to the true zeroerror rate as n increases, which is generally the case withsmall example sets. In this way, it was possible to obtain theaverage rate of accuracy for each algorithm, which was usedfor weighting the final decision for EC.

The results demonstrated that the EC provides a moreprecise classification than the use of individual classifiers.This can be seen in Figure 7, which shows Boxplots foreach type of classifier employed. The right-hand box plotrefers to the results of the EC and displays a higher medianaccuracy than the other classifiers (the median values canbe seen in Table III). Moreover, a smaller dispersion of theresults obtained by the EC can be detected, which shows agreater stability in its execution. This is noticeable from theinterquartile range of the Boxplot.

We conducted a few statistical analyses to validate theresults. Initially, we used the Shapiro-Wilk method to checktheir suitability for normality hypothesis and hence, this ledto parametric or non-parametric tests. As not all the p-valuesobtained are greater than 0.05 (see Table III), we rejected thenormality hypothesis with 95% reliability. This means that thenon-parametric test is the most suitable for the next analysis.

NaiveBayes J48 FuzzyNN IBk SMO

Fig. 7. Boxplots of accuracies presented by classifiers to determine theuser’s emotion. These results were obtained using the technique k-fold cross-validation with k = 10.

TABLE IIIMEDIAN (%) ACCURACIES AND THE p-VALUES OF THE EVALUATED

ALGORITHMS.

Classifiers Accuracy Shapiro WilkAverage (%) p-values

NayveBayes 59,16 0,115J48 59,77 0,076

FuzzyNN 57,21 0,984IBk 58,41 0,181

SMO 70,29 0,007Ensemble 82,53 0,594

The pairwise comparisons made with the aid of theWilcoxon Rank Sum test are shown in Table IV. The p-valuesobtained by means of the Wilcoxon technique indicate thatNaive Bayes classifiers, J48, FuzzyNN and IBk do not showa statistically significant difference in the rating of emotions.However, the classification performed by SMO (in addition tothe higher accuracy shown in Figure 7) shows a statisticallysignificant difference from the other individual classifiers.Finally, it is clear that the results obtained with the EC have astatistically significant difference from all the other individualclassifiers.

TABLE IVP-VALUES OF THE PAIR-WISE COMPARISON WITH THE WILCOXON RANKSUM TEST. VALUES BELOW 0.05 INDICATE STATISTICALLY SIGNIFICANT

DIFFERENCE BETWEEN THE RESULTS OF GROUPS.

NayveBayes J48 FuzzyNN IBk SMOJ48 1,000 - - - -

FuzzyNN 1,000 0,783 - - -IBk 1,000 1,000 1,000 - -

SMO 0,017 0,017 0,017 0,017 -Ensemble 0,003 0,003 0,003 0,003 0,003

These results demonstrate that, of all the individual classi-fiers employed, SMO is the technique that achieves the greatestaccuracy in emotion rating. However, it is outperformed whenthe Ensemble of Classifiers is used, since this approach

achieves a greater degree of accuracy than the others, whichis evidence of its efficiency in identifying emotions. Finally,Figure 8 shows the error matrix, also known as the confusionmatrix, in which the columns represent the reference data,while the rows represent the classification generated by theEC.

Happy

Happy

Disgust

Disgust

Fear

Fear

Neutral

Neutral

Anger

Anger

Surprise

Surprise

Sad

Sad

426

17

5

1

8

1

7

6

366

8

8

54

1

16

4

5

340

35

7

13

24

1

10

34

326

25

2

47

3

29

10

13

312

3

31

1

2

13

4

2

423

4

2

17

35

59

37

2

316 50

100

150

200

250

300

350

400

Fig. 8. Confusion matrix regarding the classification by the Ensemble.

Table V, generated from the Confusion Matrix, showsthe relationship between cost (False Positive (FP) rate) andsensitivity (True Positive (TP) Rate) reached by the Ensembleon the classification of each emotion.

TABLE VDETAIL CLASSIFICATION COST PRESENTED BY THE ENSEMBLE FOR EACH

EMOTION.

Class TP Rate FP RateHappy 0.957 0.012

Disgust 0.822 0.029Fear 0.764 0.028

Neutral 0.732 0.038Anger 0.701 0.028

Surprise 0.950 0.008Sad 0.710 0.048

It is worth noting that the model based on the Ensemble pro-vides a better classification of the emotions “Joy”, “Disgust”and “Surprise”. However, despite achieving a lower degree ofaccuracy than the other emotions, the Ensemble model classi-fies the other emotions as having the lower computational costand the model has a better rate of accuracy for classificationthan the models that are based on individual classifiers.

V. CONCLUSION

According to psychologists, emotions play a crucial rolewhile a patient is trying to recover from a wide range ofdiseases. Hence, we believe that this is an important featureto measure while monitoring patients. As a result, we adoptedan approach that took into consideration all these features andconducted experiments to validate the whole idea in terms ofperformance, accuracy and statistical analysis.

This paper discussed the use of images and emotions forhelping the healthcare in smart home environments in an

automatic way through an IoT infrastructure (i.e. without theneed for human intervention to detect new features). We madeuse of images to identify the emotional state of each person.The rate of accuracy was around 82% and there was a goodconvergence obtained through our proposed Ensemble model.

As future research in this area, we plan to exploit evolution-ary mechanisms to ensure that the classification algorithms offacial expression can make progress while taking into accountadvances in the treatment of certain ailments like Parkinsonsdisease. This can be again a good challenge for our modelas it will require processing cycles and communications alongwith evolutionary approaches in a heterogeneous environment.Furthermore, it is worthwhile to mention that we plan toimprove our emotion recognition system by taking into ac-count different components, such as voice analysis and homeenvironment.

ACKNOWLEDGMENT

Author would like to thank the support from Foundation forResearch Support of the State of Sao Paulo - FAPESP (processID 2016/14267-7 and 2017/21054-2) for funding the bulk ofthis research project. In addition, thank Professor MichaelBiehl of the University of Groningen and Professor Jo Ueyamaof the University of Sao Paulo.

REFERENCES

[1] E. Romero, A. Araujo, J. M. Moya, J.-M. de Goyeneche, J. C. Vallejo,P. Malagon, D. Villanueva, and D. Fraga, “Image processing basedservices for ambient assistant scenarios,” in Distributed Computing,Artificial Intelligence, Bioinformatics, Soft Computing, and AmbientAssisted Living, S. Omatu, M. P. Rocha, J. Bravo, F. Fernandez,E. Corchado, A. Bustillo, and J. M. Corchado, Eds., vol. 5518.Springer Berlin Heidelberg, 2009, pp. 800–807. [Online]. Available:https://doi.org/10.1007/978-3-642-02481-8 121

[2] L. Mano, M. Funes, T. Volpato, and J. Neto, “Explorando tecnologiasde iot no contexto de health smart home: uma abordagem para detecode quedas em pessoas idosas,” Journal on Advances in Theoretical andApplied Informatics, vol. 2, no. 1, pp. 46–57, 2016. [Online]. Available:https://doi.org/10.26729/jadi.v2i1.1667

[3] J. E. Salem, A. M. Kring, and S. L. Kerr, “More evidencefor generalized poor performance in facial emotion perception inschizophrenia.” Journal of abnormal psychology, vol. 105, no. 3,p. 480, 1996. [Online]. Available: http://dx.doi.org/10.1037/0021-843X.105.3.480

[4] M. K. Mandal, R. Pandey, and A. B. Prasad, “Facial expressions ofemotions and schizophrenia: A review.” Schizophrenia bulletin, vol. 24,no. 3, p. 399, 1998.

[5] R. E. Gur, C. G. Kohler, J. D. Ragland, S. J. Siegel, K. Lesko,W. B. Bilker, and R. C. Gur, “Flat affect in schizophrenia: Relationto emotion processing and neurocognitive measures,” SchizophreniaBulletin, vol. 32, no. 2, pp. 279–287, 2006. [Online]. Available:http://dx.doi.org/10.1093/schbul/sbj041

[6] B. L. Fredrickson and R. W. Levenson, “Positive emotions speedrecovery from the cardiovascular sequelae of negative emotions,”Cognition and Emotion, vol. 12, no. 2, pp. 191–220, 1998. [Online].Available: https://doi.org/10.1080/026999398379718

[7] G. V. Ostir, I.-M. Berges, M. E. Ottenbacher, A. Clow, and K. J.Ottenbacher, “Associations between positive emotion and recovery offunctional status following stroke,” Psychosomatic medicine, vol. 70,no. 4, p. 404, 2008.

[8] V. Rialle, F. Duchene, N. Noury, L. Bajolle, and J. Demongeot,“Health” smart” home: information technology for patients at home,”Telemedicine Journal and E-Health, vol. 8, no. 4, pp. 395–409, 2002.[Online]. Available: http://doi.org/10.1089/15305620260507530

[9] J. C. Augusto and P. McCullagh, “Ambient intelligence: Concepts andapplications,” Computer Science and Information Systems, vol. 4, no. 1,pp. 1–27, 2007.

[10] S. Helal, W. Mann, H. El-Zabadani, J. King, Y. Kaddoura, and E. Jansen,“The gator tech smart house: a programmable pervasive space,” Com-puter, vol. 38, no. 3, pp. 50–60, March 2005.

[11] J. A. Kientz, S. N. Patel, B. Jones, E. Price, E. D. Mynatt, andG. D. Abowd, “The georgia tech aware home,” in CHI ’08 ExtendedAbstracts on Human Factors in Computing Systems, ser. CHI EA ’08.New York, NY, USA: ACM, 2008, pp. 3675–3680. [Online]. Available:http://doi.acm.org/10.1145/1358628.1358911

[12] K. R. Scherer, “What are emotions? and how can they be measured?”Social Science Information, vol. 44, no. 4, pp. 695–729, 2005. [Online].Available: https://doi.org/10.1177/0539018405058216

[13] S. Mahlke and M. Minge, Consideration of MultipleComponents of Emotions in Human-Technology Interaction.Springer Berlin Heidelberg, 2008, pp. 51–62. [Online]. Available:https://doi.org/10.1007/978-3-540-85099-1 5

[14] L. Y. Mano, E. Vasconcelos, and J. Ueyama, “Identifying emotions inspeech patterns: Adopted approach and obtained results,” IEEE LatinAmerica Transactions, vol. 14, no. 12, pp. 4775–4780, Dec 2016.

[15] L. Mano, L. Sawada, and J. Ueyama, “Abordagem para a interaoafetiva: um estudo de caso com player de msica,” Journal on Advancesin Theoretical and Applied Informatics, vol. 2, no. 2, pp. 47–54, 2016.[Online]. Available: https://doi.org/10.26729/jadi.v2i2.2108

[16] P. Ekman, Darwin and facial expression: A century of research in review.Ishk, 2006.

[17] Y.-L. Tian, T. Kanade, and J. F. Cohn, Facial Expression Analysis.Springer New York, 2005, pp. 247–275. [Online]. Available:https://doi.org/10.1007/0-387-27257-7 12

[18] L. Y. Mano, B. S. Faial, L. H. Nakamura, P. H. Gomes, G. L.Libralon, R. I. Meneguete, G. P. Filho, G. T. Giancristofaro, G. Pessin,B. Krishnamachari, and J. Ueyama, “Exploiting iot technologies forenhancing health smart homes through patient identification and emotionrecognition,” Computer Communications, vol. 89-90, pp. 178–190, 2016.[Online]. Available: https://doi.org/10.1016/j.comcom.2016.03.010

[19] J. M. Saragih, S. Lucey, and J. F. Cohn, “Deformable model fitting byregularized landmark mean-shift,” International Journal of ComputerVision, vol. 91, no. 2, pp. 200–215, Jan 2011. [Online]. Available:https://doi.org/10.1007/s11263-010-0380-4

[20] L. Y. Mano, G. T. Giancristofaro, B. S. Faical, G. L. Libralon,G. Pessin, P. H. Gomes, and J. Ueyama, “Exploiting the use ofensemble classifiers to enhance the precision of user’s emotionclassification,” in Proceedings of the 16th International Conference onEngineering Applications of Neural Networks (INNS), ser. EANN ’15.New York, NY, USA: ACM, 2015, pp. 5:1–5:7. [Online]. Available:http://doi.acm.org/10.1145/2797143.2797165

[21] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern classification. JohnWiley & Sons, 2012.

[22] B. Schuller, S. Reiter, R. Muller, M. Al-Hames, M. Lang, andG. Rigoll, “Speaker independent speech emotion recognition byensemble classification,” in 2005 IEEE International Conference onMultimedia and Expo, July 2005, pp. 864–867. [Online]. Available:https://doi.org/10.1109/ICME.2005.1521560

[23] J. N. Bailenson, E. D. Pontikakis, I. B. Mauss, J. J. Gross,M. E. Jabon, C. A. Hutcherson, C. Nass, and O. John, “Real-timeclassification of evoked emotions using facial feature tracking andphysiological responses,” International Journal of Human-ComputerStudies, vol. 66, no. 5, pp. 303 – 317, 2008. [Online]. Available:http://www.sciencedirect.com/science/article/pii/S1071581907001462

[24] G. Yuan, T. S. Lim, W. K. Juan, H. M.-H. Ringo, and Q. Li,“A gmm based 2-stage architecture for multi-subject emotionrecognition using physiological responses,” in Proceedings of the 1stAugmented Human International Conference, ser. AH ’10. NewYork, NY, USA: ACM, 2010, pp. 3:1–3:6. [Online]. Available:http://doi.acm.org/10.1145/1785455.1785458

[25] K. Faceli, A. C. Lorena, J. Gama, and A. Carvalho, “Inteligenciaartificial: Uma abordagem de aprendizado de maquina,” Rio de Janeiro:LTC, vol. 2, p. 192, 2011.

[26] C. Peter and B. Urban, Emotion in Human-Computer Interaction,J. Dill, R. Earnshaw, D. Kasik, J. Vince, and P. C. Wong, Eds. London:Springer London, 2012. [Online]. Available: https://doi.org/10.1007/978-1-4471-2804-5 14

[27] S. Ramakrishnan and I. M. M. El Emary, “Speech emotion recognitionapproaches in human computer interaction,” TelecommunicationSystems, vol. 52, no. 3, pp. 1467–1478, Mar 2013. [Online]. Available:https://doi.org/10.1007/s11235-011-9624-z

[28] G. Littlewort, J. Whitehill, T. Wu, I. Fasel, M. Frank, J. Movellan, andM. Bartlett, “The computer expression recognition toolbox (cert),” inFace and Gesture 2011, March 2011, pp. 298–305. [Online]. Available:https://ieeexplore.ieee.org/document/5771414/

[29] C. D. Katsis, N. Katertsidis, G. Ganiatsas, and D. I. Fotiadis, “Towardemotion recognition in car-racing drivers: A biosignal processingapproach,” IEEE Transactions on Systems, Man, and Cybernetics -Part A: Systems and Humans, vol. 38, no. 3, pp. 502–512, May 2008.[Online]. Available: https://ieeexplore.ieee.org/document/4490039/

[30] D. Ro and H. Pe, Pattern Classification and Scene Analysis, Wiley, Ed.Wiley, 1973.

[31] G. Chanel, J. J. Kierkels, M. Soleymani, and T. Pun,“Short-term emotion assessment in a recall paradigm,”International Journal of Human-Computer Studies, vol. 67,no. 8, pp. 607 – 627, 2009. [Online]. Available:http://www.sciencedirect.com/science/article/pii/S1071581909000433

[32] G. Coulson, G. Blair, P. Grace, F. Taiani, A. Joolia, K. Lee, J. Ueyama,and T. Sivaharan, “A generic component model for building systemssoftware,” ACM Trans. Comput. Syst., vol. 26, no. 1, pp. 1:1–1:42, 2008.

[33] O. Langner, R. Dotsch, G. Bijlstra, D. H. J. Wigboldus, S. T. Hawk,and A. van Knippenberg, “Presentation and validation of the radboudfaces database,” Cognition and Emotion, vol. 24, no. 8, pp. 1377–1388,2010. [Online]. Available: https://doi.org/10.1080/02699930903485076

[34] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews,“The extended cohn-kanade dataset (ck+): A complete dataset for actionunit and emotion-specified expression,” in 2010 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition - Workshops,June 2010, pp. 94–101.

[35] J. P. Mena-Chalco, L. Velho, and R. C. Junior, “3d human facereconstruction using principal components spaces,” in Proceedings ofWTD SIBGRAPI Conference on Graphics, Patterns and Images. IMPA-RJ, Tech. Rep, 2011, pp. 1–6.

[36] N. C. Ebner, M. Riediger, and U. Lindenberger, “Faces—a databaseof facial expressions in young, middle-aged, and older womenand men: Development and validation,” Behavior Research Methods,vol. 42, no. 1, pp. 351–362, Feb 2010. [Online]. Available:https://doi.org/10.3758/BRM.42.1.351

[37] R. R. Bouckaert, E. Frank, M. Hall, R. Kirkby et al., “Weka manual forversion 3-7-8,” 2013.