Information Sciences 272 (2014) 16–28

Contents lists available at ScienceDirect

Information Sciences

journal homepage: www.elsevier .com/locate / ins

Unsupervised product feature extraction for feature-orientedopinion determination

http://dx.doi.org/10.1016/j.ins.2014.02.0630020-0255/� 2014 Elsevier Inc. All rights reserved.

⇑ Corresponding author. Tel.: +86 13003006323.E-mail addresses: quanchqin@hfut.edu.cn (C. Quan), ren@is.tokushima-u.ac.jp (F. Ren).

Changqin Quan a,⇑, Fuji Ren a,b

a AnHui Province Key Laboratory of Affective Computing and Advanced Intelligent Machine, School of Computer and Information, HeFei University of Technology,Hefei 230009, Chinab Faculty of Engineering, University of Tokushima, 2-1 Minami-Josanjima, Tokushima 770-8506, Japan

a r t i c l e i n f o a b s t r a c t

Article history:Received 5 March 2013Received in revised form 20 January 2014Accepted 9 February 2014Available online 19 February 2014

Keywords:Product feature extractionSentiment analysisDomain corporaTerm similarityOpinion lexicon

Identifying product features from reviews is the fundamental step as well as a bottleneck infeature-level sentiment analysis. This study proposes a method of unsupervised productfeature extraction for feature-oriented opinion determination. The domain-specific fea-tures are extracted by measuring the similarity distance of domain vectors. A domain vec-tor is derived based on the association values between a feature and comparative domaincorpora. A novel term similarity measure (PMI–TFIDF) is introduced to evaluate the asso-ciation of candidate features and domain entities. The results show that our approach offeature extraction outperforms other state-of-the-art methods, and the only externalresources used are comparative domain corpora. Therefore, it is generic and unsupervised.Compared with traditional pointwise mutual information (PMI), PMI–TFIDF showed betterdistinction ability. We also propose feature-oriented opinion determination based on fea-ture-opinion pair extraction and feature-oriented opinion lexicon generation. The resultsdemonstrate the effectiveness of our proposed method and indicate that feature-orientedopinion lexicons are superior to general opinion lexicons for feature-oriented opiniondetermination.

� 2014 Elsevier Inc. All rights reserved.

1. Introduction

With the growing popularity and availability of user-generated reviews, recently increasing more attention is being paidto the research of sentiment analysis (SA). SA is also called opinion mining, and aims at determining the attitude polarity(positive, negative or neutral) of a text. The potential benefits of SA on education, e-commerce, and opinion polls are signif-icant. Consequently, a variety of techniques have performed to probe SA. SA techniques basically include two branches: doc-ument-level SA and feature-level SA. Document-level SA focuses on determining an overall opinion for a document.Representative work includes Turney applied an unsupervised method for detecting the polarity of product reviews [46];Pang applied machine learning techniques for sentiment classification of movie reviews [29]. The following researches onpredicting star ratings on different scales extended the basic task of classifying a review as either positive or negative[30,35,39,53].

Feature-level SA aims to extract features (e.g., camera’s image quality, size) from reviews, and then determine opinionsthat are linked with each feature. Compared with document-level SA, feature-level SA is able to provide more fine grained

C. Quan, F. Ren / Information Sciences 272 (2014) 16–28 17

sentiment analysis on certain opinion targets, and has a wider range of applications [24,49]. Representative work in feature-level SA includes Hu and Liu proposed a method for mining and summarizing product reviews [17]; Popescu and Etzioni ex-tended Hu and Liu’s method through web search [31]; Ding et al. proposed a holistic lexicon-based approach to improve Huand Liu’s method [10].

Identifying product features from reviews is the fundamental step as well as a bottleneck in the problem of feature-levelSA. Although several researchers have studied opinion lexicon expansion and opinion product feature extraction problems,their algorithms either need additional and external resources or impose strong constraints, and are of limited success [33].

As SA is sensitive to the domain, product feature extraction can be seen as a problem of domain-specific entity recogni-tion. However, most of the existing methods toward domain-specific entity recognition often rely on domain-specific knowl-edge to improve system performance, but such knowledge is expensive to build and maintain for various domains, and adomain dependent method is hard to extend to other domains. Therefore, extracting domain-specific product feature in ageneric and unsupervised manner would be desirable for feature-level SA.

In this paper, we propose an approach that exploits domain-specificity of words as a form of domain-knowledge for do-main-specific feature extraction. The basic idea of our approach is to extract domain product features through the evaluationof their weights in different domains. Compared to prior work, our approach is generic and unsupervised. In evaluation, wecompare the proposed method with several state-of-the-art methods (including unsupervised and semi-supervised) using astandard product review test collection. The experimental results demonstrate that our approach outperforms other unsu-pervised and semi-supervised methods.

Furthermore, we explore the problem of product feature opinion determination, which includes three main subtasks: (1)feature-opinion pair extraction; (2) feature-oriented opinion lexicon generation; and (3) feature-oriented opinion determi-nation. We compare the results with different experimental setups, and the experiments confirm the effectiveness of ourproposed feature-opinion pair extraction method, and indicate that feature-oriented opinion lexicons are superior to generalopinion lexicons for feature-oriented opinion determination.

The remainder of this paper is organized as follows. In Section 2, a literature review is conducted on the studies of auto-matic product feature extraction and product feature opinion determination. Section 3 presents the proposed method fordomain-specific product feature extraction using comparative domain corpora. Section 4 describes feature-oriented opiniondetermination. Section 5 presents the evaluation and discussion. Section 6 concludes this work with closing remarks andfuture directions.

2. Literature review

Feature-level SA is essentially a content based information extraction and classification problem involving the main stepsof feature extraction, relation extraction, opinion classification, etc. This study focuses on two fundamental problems includ-ing product feature extraction and product feature opinion determination.

2.1. Product feature extraction

Previous studies have explored many different methods for this problem. In [17], a frequency-based approach was pro-posed to find frequent nouns and noun phrases, and then opinion words were used to extract infrequent aspects based onthe idea of ‘‘opinions have targets’’. This idea was generalized to a double propagation method [33] and a dependency basedmethod [34]. Popescu and Etzioni extended Hu and Liu’s method by computing the PMI score between phrase and class-specified discriminators through a web search [31]. The necessary of web querying in this work has been pointed out aproblem [5]. Ding and Liu further improved Hu and Liu’s system by adding some rules to handle different kinds of sentencestructures [9]. However, sentence structure rules are language dependent, so it is hard to be applied to other languages.

Jin et al. proposed a supervised learning method based on lexicalized Hidden Markov Models (L-HMMs) that integratesseveral linguistic features [19]. The authors demonstrated that their method is more effective and accurate than rule basedmethod. The main problem is that SA is sensitive to the domain of training data. A classifier trained using reviews from onedomain often performs poorly in another domain.

Scaffidi et al. applied a language model approach with the assumption that product features are mentioned more often ina product review than they are mentioned in generic English [38]. But the results may sensitive on corpus size.

Somasundaran and Wiebe used syntactic dependency for aspect and opinion extraction [40]. Ma and Wan exploitedaspects extraction in the news title and contents [25]. Yu et al. explored important aspects extraction [51]. More surveycan be found in [24].

As SA is sensitive to domains, product feature extraction can be seen as a problem of domain-specific entity recognition.Named entity recognition (NER) is a preprocessing tool to many natural language processing tasks, such as text summariza-tion [2], machine translation, and document classification [1,36]. Traditional NER task has expanded beyond identifying peo-ple, location, and organization to book titles, email addresses, phone numbers, and protein names [28]. Recently there hasbeen a surge of interest in extracting product attributes from online reviews. Recent work on product attribute extraction by[4] applied a Latent Dirichlet Allocation (LDA) model to identify different features of products from user reviews. Veselin andCardie treated feature extraction as a topic coreference resolution problem, and they proposed to train a classifier to judge if

18 C. Quan, F. Ren / Information Sciences 272 (2014) 16–28

two opinions are on the same feature [47]. Mei et al. also applied topic modeling to capture features in reviews [26]. Topicmodeling is to model the generation of a document set and to mine the implied topics in the documents. But it may generatefeatures that are vague and not easily interpretable. Indeed, how to refine discovered aspects and clean up words in eachaspect remains an open question. Putthividhya and Hu presented a bootstrapped algorithm that can identify new brandnames corresponding to spelling invariants or typographical errors of the known brands in the seed list and novel brandsor designers [32]. Their approach is supervised.

There are still many methods for domain-specific NER have been proposed over the past years. For example, Giouli et al.focused on building a homogenous, reusable and adaptable linguistic resource to different domains and languages [13], butsuch knowledge is expensive to build and maintain for various domains. Jiang and Zhai explored the domain structure intraining data for domain-specific entity recognition [18]. Zhang and Iria proposed unsupervised term extraction to computedomain-specificity of words, and applied a feature extraction technique to derive useful features as domain-specific knowl-edge for NER [52]. However, most of the work has concentrated on limited domains, and few studies explored the use ofcomparative domain corpora for the problem of domain-specific product feature extraction.

2.2. Product feature opinion determination

Opinion words such as ‘‘good’’, ‘‘bad’’ are basic discriminators for opinion determination. Extensive work has been doneon opinion word extraction and word polarity determination. Representative work includes the use of conjunction that putsconstraints on conjoined adjectives to detect adjectives with positive and negative orientation by Hatzivassiloglou andMcKeown [14], Turney and Littman’s method for inferring the semantic orientation of a word from its statistical associationwith a set of positive and negative paradigm words measured by pointwise mutual information (PMI) and latent semanticanalysis (LSA) [45], Wiebe et al.’s method that used distributional similarity to identify the cues of subjectivity [50]. Further-more, many new similarity methods have been proposed, for example, Hossein Zadeh and Reformat proposed to assesssemantic similarity of concepts defined in ontology [16], Mirkina and Nascimentoc proposed an additive spectral methodfor similarity analysis [27], Tapia-Roseroa et al. proposed a method based on shape-similarity for detecting similar opinionsin group decision-making [44], and Wang proposed a neighborhood similarity method for distance measurement [48].

As described previously, SA is sensitive to the domain, so product feature opinion words are certainly domain dependent.Recently several methods have been proposed on domain dependent opinion words extraction. Chetviorkin and Loukachev-itch proposed a supervised term classification method for domain-specific opinion word extraction [6]. Their method sup-posed that some domains have similar sentiment lexicons and utilized this fact to build an opinion word vocabulary for agroup of domains. Cruz et al. proposed a domain-specific, resource-based opinion extraction system, which extracts opinionwords from annotated corpus. For each opinion word, the measures of support, opinion word probability, feature-basedopinion word probabilities, and feature-based semantic orientations are evaluated [7]. The main problem of the abovetwo proposals is that their work required annotated corpus. The construction can be both time-consuming and expensive.

Different from annotated corpus based method, Qiu et al. took advantage of domain dependent corpora and two contex-tual evidence based observations: ‘‘same polarity for same target in a review’’ and ‘‘same polarity for same opinion word in adomain corpus’’ for opinion word polarity assignment [33]. But their method needs an initial opinion lexicon as seeds, whichwere provided manually. Consequently, the generality is a problem.

Lexicon-based methods for opinion determination have received a lot of attention in opinion analysis task [8,15,20,43].And there are many lexical resources for these tasks, such as the General Inquirer (GI) [41], WordNet-Affect [42], NTU Sen-timent Dictionary [21], Hownet [11], Senti-Wordnet [12]. The biggest shortcoming of lexicon-based methods is that they arehard to find domain dependent opinion words, and Qiu also addressed this issue [34]. Our work proposes to combine generalsentiment lexicon and domain dependent corpora. Therefore we can find domain dependent opinion words without loss ofgenerality.

3. Product feature extraction using comparative domain corpora

The product feature extraction method proposed in this paper is based on comparative domain corpora. Comparative do-main corpora are several product review sets. The basic idea of our approach is to extract domain product features throughthe evaluation of their weights in different related domains. The association computation between features and domains isthe key to the extraction. We apply a term similarity measure to evaluate the association of candidate features and domainwords. Based on these similarities, a domain vector for each candidate feature can be derived. Then the domain-specific fea-tures are extracted by measuring the distances between the domain vectors of features and the domain vector of a domainentity.

3.1. Deriving domain vectors for candidate features

The hypothesis of this approach is that, in a certain domain review corpus, a domain-specific feature has closer associa-tion with the domain entity of the current corpus than with another domain entity of a comparative domain corpus. Forexample, the semantic of the feature ‘‘photo quality’’ has closer association with the domain entity ‘‘camera’’ than with

Fig. 1. The process of deriving domain vectors for candidate features.

Table 1The symbols used in Fig. 1 and their descriptions.

Symbol Descriptions

DRC Domain review corpus: A review set of a certain domainDRCt: Domain review corpus that used for deriving domain-specific featuresDRC1� � �DRCn: Comparative domain review corpora

DE Domain entity: A term that represents a certain domain (e.g., camera, computer)CF Candidate feature: Candidate features that derived from DRC through the steps of phrase recognition and term extractionDV Domain vector: Each dimension of a domain vector records the similarity between a candidate feature and a domain entity Sim(Cfi, DEj)

C. Quan, F. Ren / Information Sciences 272 (2014) 16–28 19

‘‘mp3’’ in a camera review corpus. Consequently, besides the given domain review corpus (DRCtin Fig. 1) that used for deriv-ing domain-specific features, several comparative domain review corpora (DRC1� � �DRCn in Fig. 1) are required. We will dis-cuss the effects of using different comparative domain review corpora on the performance of feature extraction in theexperiment section.

Fig. 1 illustrates the process of deriving domain vectors for candidate features. Table 1 lists the symbols used in Fig. 1 andtheir descriptions.

As domain-specific features are often nouns and noun phrases, nouns and multiple nouns string are extracted in the pre-processing step. The candidate feature extraction step includes noun phrase extraction, name entity recognition and stopwords filtration. In this work, we employ the Stanford POS tagging tool1 to do the POS tagging.

For each domain review corpus (DRC), we use a domain entity (DE) term to represent it. For example ‘‘camera’’ can be a DEfor digital camera review corpus; ‘‘phone’’ can be a DE for cell phone review corpus. The association between a feature and adomain is embodied by the similarity between the feature word and the DE term. The similarity is evaluated by a new mea-sure of PMI–TFIDF, which combines pointwise mutual information (PMI) and term frequency–inverse document frequency(TF–IDF). A desired DE should meet two conditions: a noun and with high frequency in a domain corpus.

For each candidate feature, a domain vector is derived based on the similarity between the feature word and the DE term.The similarity reflects its relevance to the comparative domain corpora. The dimension of a domain vector (DV) is the num-ber of comparative domain corpora.

3.2. Measuring the association of candidate features and domains

As stated previously, the association computation between features and domains is the key to feature extractions. In thissection, we will describe the association computation in detail.

There are two problems should be considered for measuring the association of candidate features and domains:

(1) How to evaluate the association of candidate features and domain words (Sim(Cfi, DEj) in Fig. 1)?(2) How to measure the association of candidate features and domains based on domain vector (DV)?

For the problem (1), we apply a term similarity measure to evaluate the association of candidate features and domainwords. The value of Sim(Cfi, DEj) in Fig. 1 is measured by a new measure of PMI–TFIDF which combines pointwise mutualinformation (PMI) and term frequency–inverse document frequency (TF–IDF). PMI [3] is a measure of association used in

1 http://nlp.stanford.edu/software/tagger.shtml.

20 C. Quan, F. Ren / Information Sciences 272 (2014) 16–28

information theory, which has been applied in many sentiment analysis methods to measure the association of words, seeEq. (1)

PMIðwi;wjÞ ¼ logPrðwi;wjÞ

PrðwiÞPrðwjÞð1Þ

where Pr (wi, wj) is the probability of a sentence containing word wi and word wj in a corpus, Pr (w) is the probability of asentence containing word w in a corpus.

The statistical estimation would be unreliable if the corpus is small, that is a common problem for many statistical ap-proaches. This is no exception to PMI and the main problem of PMI for measuring the association of words is that PMI valueis sensitive to corpus size. In a small corpus, different words tend to get the same PMI value, and thus it is hard to distinguishthe degree of similarity. So we combine TF–IDF with PMI to evaluate the association of candidate features and domain words.

The TF–IDF weight (term frequency–inverse document frequency) [37] is a numerical statistic which reflects how impor-tant a word is to a document in a corpus, see Eqs. (2)–(4).

TFij ¼nijPknkj

ð2Þ

where nij is the frequency of word i in document j,P

knkj is the number of all words in document j.

IDFi ¼ logjDj

jd : wi 2 dj ð3Þ

where |D| is the number of documents in a corpus, |d:wi e d| is the number of documents containing word wi.The importance of word i to document j can be weighted by Eq. (4).

wij ¼ TFij � IDFi ð4Þ

The combination of PMI and TF–IDF is represented by Eq. (5).

PMI—TFIDFðwi;wjÞ ¼ logPrðwi;wjÞ �

PkTFIDFðwi; dkÞ �

PkTFIDFðwj;dkÞ

PrðwiÞPrðwjÞð5Þ

whereP

kTFIDF(wi, dk) is the sum of word i to all documents in the corpus.P

kTFIDFðwi; ;dkÞ� �

andP

kTFIDF(wj, dk) add theimportance measure of word i and word j to all documents in the corpus.

By combining PMI and TF–IDF, the main problem of PMI can be solved effectively. With the association measured by PMI–TFIDF, a domain vector for each candidate feature can be derived. Each dimension of a domain vector represents the degreeof association between a candidate feature and a domain entity.

For the problem (2), we apply a vector similarity measure to evaluate the association of candidate features and domainsbased on domain vector (DV).

As domain entity (DE) is one of the candidate features (Cf) as well, and satisfies DEt e {Cf1, � � � , Cfm}, we evaluate each can-didate feature by measuring the distance between its domain vector and the domain vector of a domain entity. Based on thecosine similarity between two vectors, we use Eq. (6) to get the similarity between two domain vectors.

SimðDVcf ;DVdeÞ ¼Pn

i¼1ðDVcfi � DVdeiÞffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPn

i¼1DV2cfi

q�ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPn

i¼1DV2dei

q ð6Þ

where DVcf is the domain vector of a candidate feature, DVde is the domain vector of a domain entity. If the value ofSim(DVcf, DVde) above the threshold value a, cf would be recognized as a domain-specific feature dsf, see Eq. (7).

cf 2 fdsfg if SimðDVcf ;DVdeÞ > a ð7Þ

This approach requires no additional resources except several comparative domain review corpora, which are readilyavailable.

4. Feature-oriented opinion determination

Feature-oriented opinion determination includes three main subtasks: (1) feature-opinion pair extraction; (2) feature-oriented opinion lexicon generation; and (3) feature-oriented opinion determination.

4.1. Feature-opinion pair extraction

Feature-opinion pair extraction is the key for both feature-oriented opinion lexicon generation and feature-oriented opin-ion determination. Given a set of features F = {f1, f2, � � � , fn} of a certain product, the main goal of feature-opinion pair extrac-tion is to identify the associated opinion word set Oi = {oi1, oi2, � � � , oim} for each feature fi e F. The extracted feature-opinionpair has the following form: (fi, Oi).

C. Quan, F. Ren / Information Sciences 272 (2014) 16–28 21

We take the following steps to extract feature-opinion pair:

Step I: Extracting feature-opinion pair by using dependency parse. Based on the definition of direct and indirect depen-dency given by [30], we define dependency distance (dd) between a feature fi and a potential opinion word oj by Eq. (8).

2 http

ddðfi; ojÞ ¼1 if there is a direct dependency between f i and oj;

dependency numðfi ! ojÞ if there is an indirect dependency between f i and oj;

þ1 otherwise

8><>: ð8Þ

where dependency_num(fi ? oj) is the minimum number of dependency relations from fi (or oj) to oj (or fi) in the dependencyrelations of a sentence.

In this work, we consider potential opinion words to be adjectives. If dd(fi, oj) is less than threshold value b, oj would beconsidered a candidate opinion word for feature fi.

Step II: Filtering candidate opinion word for feature fi based on the value of PMI–TFIDF(fi, oj) (see Eq. (5)). If PMI–TFIDF(fi, oj)is less than the threshold value c, oj would be filtered. Unfiltered opinion words would be added to the opinion wordset Oi.

4.2. Feature-oriented opinion lexicon generation

Based on the results of feature-opinion pair extraction, the main task of feature-oriented opinion lexicons extraction isdetermining the polarity of each opinion word in the associated opinion word set Oi = {oi1, oi2, � � � , oim} for feature fi andderiving feature-oriented positive and negative opinion lexicons.

We propose to combine general sentiment lexicon (General Inquirer [41]) and domain dependent corpora for feature-ori-ented opinion lexicon generation. The General Inquirer (GI) is a lexicon attaching syntactic, semantic, and pragmatic infor-mation to part-of-speech tagged words2. We collect the words with ‘‘Positiv’’ and ‘‘Negativ’’ labels from GI separately as GIpositive lexicon and GI negative lexicon. GI positive lexicon contains 1636 words, and GI negative lexicon contains 2005 words.

Given a feature fi and its associated opinion word set Oi = {oi1, oi2, � � � , oim}, the polarity of an opinion word oij is assignedby Eq. (9).

opinion polarityðoijÞ ¼positive ; if oij 2 GI-posnegative ; if oij 2 GI-negneutral ; otherwise

8><>: ð9Þ

where oij e Oi, GI-pos and GI-neg are GI positive lexicon and GI negative lexicon. We can derive three lexicons for the featurefi: pos(fi), neg(fi), and neutral(fi).

Inspired by the basic observation of the opinion word polarity assignment method proposed by Qiu [33] ‘‘same polarityfor same feature in a review’’, we further extend pos(fi) and neg(fi) from neutral(fi) by Eq. (10).

opinion polarityðoijÞ ¼positive ; if oik 2 posðfiÞ and oij [ oik 2 rnegative ; if oip 2 negðfiÞ and oij [ oip 2 r

�ð10Þ

where oij, oik and oip are associated opinion words for feature fi, oij e neutral(fi), r is a review. oij [ oik e r means that oij and oik

appear in the same review. oij would be skipped if there is a negation word appears in the surrounding 3-word window of oij

in review r.

4.3. Feature-oriented opinion determination

Kernel methods (KMs) are a class of algorithms for pattern analysis, whose best known element is the Support VectorMachine (SVM), which had been well used in many applications. In this work, we apply KM for feature-oriented opiniondetermination.

In the basic vector-space model, documents are represented by a matrix D, whose columns are indexed by documents androws are indexed by terms. The corresponding kernel is given by the inner product between the feature vectors, see Eqs. (11)and (12).

K ¼ D0D ð11Þ

kðd1;d2Þ ¼ h/ðd1Þ;/ðd2Þi ¼XN

j¼1

tf ðtj;d1Þtf ðtj;d2Þ ð12Þ

://www.wjh.harvard.edu/~inquirer/spreadsheet_guide.htm.

22 C. Quan, F. Ren / Information Sciences 272 (2014) 16–28

Document d is represented by a row vector, see Eq. (13).

Table 2Charact

Data

D1D2D3D4D5D6D7D8D9D10

/ðdÞ ¼ ðtf ðt1; dÞ; . . . ; tf ðtj;dÞ; . . . ; tf ðtN;dÞÞ 2 RN ð13Þ

where tf(ti, d) is the frequency of term ti appeared in document d.A linear transformation is /(d) = /(d) � S, where S is any appropriately shaped matrix, can be set by Eq. (14).

S ¼ RP ð14Þ

where R is a term-weight matrix, and is diagonal, whose entry R(i, i) is the weight of the term i, can be defined by the inversedocument frequency idf(t) = ln (l/df(t)), l is the total number of documents in the corpus, df(t) is the number of documentsthat contain the given term t. P is a term-document matrix. The new kernel K for this feature space is defined by Eq. (15).

K ¼ D0D ¼ ðDSÞ0DS ¼ ðDRPÞ0DRP ð15Þ

With the basic kernel, we can derive various kernels. For example, Latent semantic kernel (LSK) can be produced by sin-gular value decomposition (SVD) of the term-document matrix.

Similarly, the derived polynomial kernel (PK) is defined by Eq. (16).

kðd1; d2Þ ¼ ðkðd1;d2Þ þ cÞd ð16Þ

The derived Gaussian Kernel (GK) is defined by Eqs. (17) and (18).

kðd1; d2Þ ¼ exp � jjd1 � d2jj2r2

� �ð17Þ

kðd1; d2Þ ¼ exp � kðd1; d1Þ � 2kðd1; d2Þ þ kðd2;d2Þ2r2

� �ð18Þ

Our previous work has demonstrated that PK has higher precision and efficiency than LSK and GK for the problem of sen-timental classification [35]. Therefore, in this work, we apply PK for feature-oriented opinion determination.

We firstly group the reviews according to the features they contain, and build PK for each feature using the generatedfeature-oriented opinion lexicons. The polarity of feature fi in review rt is determined by Eq. (19).

opinion polarityðfi rtÞ ¼positive ; if simðposðfiÞ;OtðfiÞÞ > simðnegðfiÞ;OtðfiÞÞnegative ; if simðposðfiÞ;OtðfiÞÞ < simðnegðfiÞ;OtðfiÞÞneutral ; if simðposðfiÞ;OtðfiÞÞ ¼ simðnegðfiÞ;OtðfiÞÞ

8><>: ð19Þ

where fi rt represents feature fi in review rt, Ot(fi) is the associated opinion word set for feature fi in review rt. pos(fi) andneg(fi) are feature-oriented opinion lexicons for feature fi. simðposðfiÞ;OtðfiÞÞ is the similarity value of pos(fi) and Ot(fi) com-puted by PK. Similarly, simðnegðfiÞ;OtðfiÞÞ is the similarity value of neg(fi) and Ot(fi). opinion_polarity(fi rt) would be reversedif there is a negation word appears in the surrounding 3-word window of an opinion word for feature fi.

5. Experiments and discussions

5.1. Dataset

In our experiments, we use datasets of reviews for 10 classes (482 reviews) which include the publicly available datasetsfor four review classes from [17] and six review classes. These reviews involve four product classes: digital camera (D1, D2,D5), cell phone (D3, D10); mp3 player (D4, D7, D9), and router (D6, D8). The details of this dataset are shown in Table 2.

eristics of the review datasets.

Data description Num. of reviews Num. of features

Digital camera: Canon G3 45 106Digital camera: Nikon coolpix 4300 34 72Cell phone: Nokia 6610 41 109Mp3 player: Creative Labs Nomad Jukebox Zen Xtra 40GB 95 184Digital camera: Canon S100 50 109Router: Hitachi router 30 97Mp3 player: ipod 44 104Router: Linksys Router 47 104Mp3 player: MicroMP3 47 220Cell phone: Nokia 6600 49 178

C. Quan, F. Ren / Information Sciences 272 (2014) 16–28 23

5.2. Experiments on product feature extraction

Table 3 shows the experimental results of extracting domain-specific features on 10 review classes. For each review class,ten review datasets are used as comparative domain corpora. The performances are measured using the standard evaluationmeasures of precision (p), recall (r) and F-score (f), f = 2pr/(p + r). The threshold value a (see Eq. (7)) is empirically set.

Using the same ten review datasets as comparative domain corpora, Table 3 compares the results of precision, recall, andF-score on the four domains. The best results were on Cell phone domain; the worst results were on Router domain. In addi-tion to the differences in datasets may have different performances, different domain entities can affect the results greatly.Table 4 shows the derived domain vectors of domain entities for the four domains. d1–d10 In Table 4 are the dimension val-ues of domain vector, each dimension corresponds to a comparative domain corpus in Table 2. The domain vector of a do-main entity represents the association of a domain entity and different domains, which is evaluated by the similaritymeasure of PMI–TFIDF described in Section 3.2.

From Table 4, we can find that, for each domain entity, nonzero values appeared in the associated domain corpus. Forexample, nonzero values of the domain entity ‘‘camera’’ appeared in d1, d2, d3, d5 and d10, in which d1, d2, and d10 aredigital camera review datasets, d3 and d10 are cell phone review datasets, which also have similar characteristics with dig-ital camera, but the dimension values on d3 and d10 are relatively small. We also find that the domain entity ‘‘phone’’showed better distinction ability than other domain terms because the dimension values of its domain vector are higheron cell phone review datasets (d3 and d10). In contrast, the dimension values of domain entity ‘‘router’’ are relatively lowon router review datasets (d6 and d8). This may explain that different domain entities get different performances.

Another observation from Table 3 is that, in the same domain, the results on different datasets are quite different. As anexample of router domain, D8 has a much higher F-score, about 9% higher than D6. Further experiments told us that the useof different comparative domain review corpora can affect the performance of feature extraction. Table 5 shows the best re-sults by using different comparative domain review corpora for the four domains. The threshold values a (see Eq. (7)) areempirical values. Comparative domain corpora are gradually added based on the test dataset.

By comparing the experimental results in Table 3 and Table 5, we find that the results substantially improved by settingcomparative domain review corpora. The average F-scores of the domains on digital camera, cell phone, mp3 player, and rou-ter gained 5.2%, 2.1%, 4.2%, and 7.3% increase respectively.

Table 5 also showed that adding the corpus with the same domain to comparative domain corpora is able to improve per-formance. As an example of the digital camera domain, the best results of the three datasets (D1, D2, and D5) were achievedby mutual using of themselves as comparative domain corpora. The same situation also appeared to the domains of cellphone and router.

Table 6 shows the results comparison on precision (p), recall (r), and F-score (f) respectively of Hu and Liu [17], Popescuand Etzioni [31], Qiu et al. [33], and our approach by using the same four datasets (D1–D4).

From Table 6, we can see that on average our approach has 11% improvement in F-score over Hu and Liu [17], 5% overPopescu and Etzioni [31], 2% over Qiu et al. [33]. Compared with the above methods, the only external resources used inour method are several relative comparative domain corpora, which are available readily.

Different from Popescu and Etzioni’s method [31] which used PMI assessment to evaluate each candidate feature, inour method, we applied a new measure of PMI–TFIDF to evaluate the association of candidate features and domain words.

Table 3The experimental results of extracting domain-specific features.

Review domain Data Precision Recall F-score

Digital camera D1 0.732 0.849 0.786D2 0.772 0.847 0.808D5 0.856 0.733 0.790

Avg. 0.787 0.810 0.795

Cell phone D3 0.856 0.927 0.890D10 0.840 0.888 0.863

Avg. 0.848 0.908 0.877

Mp3 player D4 0.879 0.832 0.855D7 0.807 0.885 0.844D9 0.910 0.732 0.811

Avg. 0.865 0.816 0.837

Router D6 0.739 0.526 0.615D8 0.737 0.673 0.704

Avg. 0.738 0.600 0.660

Table 4Derived domain vectors of domain entities.

Domain entity Domain vector dimensions of domain entity

d1 d2 d3 d4 d5 d6 d7 d8 d9 d10

camera 15.2 6.3 0.4 0.0 13.0 0.0 0.0 0.0 0.0 0.6phone 0.0 0.0 19.8 0.0 0.0 0.0 0.0 0.0 0.0 15.8mp3 0.0 0.0 0.0 19.2 0.0 0.0 0.1 0.0 4.7 0.0router 0.0 0.0 0.0 0.0 0.0 4.3 0.0 8.2 0.0 0.0

Table 5The best results of extracting domain-specific features on D1–D10.

Review domain Data Comparative domain corpora (threshold a) Precision Recall F-score

Digital camera D1 D1–D7 (a = 5.0) 0.755 0.906 0.824D2 D1–D7 (a = 5.0) 0.810 0.889 0.848D5 D1–D2,D5 (a = 5.0) 0.828 0.914 0.869

Avg. 0.798 0.903 0.847

Cell phone D3 D1–D10 (a = 2.5) 0.856 0.927 0.890D10 D1–D8,D10 (a=2.5) (a=2.5) 0.878 0.933 0.905

Avg. 0.867 0.930 0.898

Mp3 player D4 D1, D4 (a = 3.0) 0.923 0.918 0.920D7 D1–D10 (a = 1.0) 0.807 0.885 0.844D9 D9 (a = 1.0) 0.853 0.895 0.873

Avg. 0.861 0.899 0.879

Router D6 D1–D4, D6 (a = 0.3) 0.848 0.691 0.761D8 D1–D10 (a = 0.5) 0.737 0.673 0.704

Avg. 0.793 0.682 0.733

Table 6Results comparison on precision (p), recall (r), and F-score (f) respectively of Hu and Liu [17], Popescu and Etzioni [31], Qiu et al [33], and our approach.

Data Hu and Liu [17](Unsupervised + WordNet)

Popescu and Etzioni [31](Unsupervised + web)

Qiu et al. [33] (Semi-supervised)

Ourapproach(unsupervised)

p r f p r f p r f p r f

D1 0.75 0.82 0.78 0.89 0.80 0.84 0.87 0.81 0.84 0.76 0.91 0.83D2 0.71 0.79 0.75 0.87 0.74 0.80 0.90 0.81 0.85 0.81 0.89 0.85D3 0.72 0.76 0.74 0.89 0.74 0.81 0.90 0.86 0.88 0.86 0.93 0.89D4 0.69 0.81 0.75 0.86 0.80 0.83 0.81 0.84 0.82 0.92 0.92 0.92

Avg. 0.72 0.80 0.76 0.88 0.77 0.82 0.87 0.83 0.85 0.84 0.91 0.87

24 C. Quan, F. Ren / Information Sciences 272 (2014) 16–28

Tables 7 and 8 show the performance comparison of measuring the association between features and domain entities by PMIand PMI–TFIDF. We use the datasets of D1, D3, D4, and D6 for the computation on the four domain entities of camera, phone,mp3, and router respectively.

The comparison between PMI and PMI–TFIDF in Tables 7 and 8 shows that the extracted features that measured by PMI–TFIDF have closer association with the domain entities than the extracted features measured by PMI. In addition, theassociation values measured by PMI–TFIDF showed a good distinction between different features. In contrast, the associationvalues measured by PMI showed a poor distinction. The reason can be seen from the definition of PMI and PMI–TFIDF (seeEqs. (1) and (5)). In a small corpus, many words have the same frequency, which means the probability Pr (dei, wi) of a sen-tence containing domain entity de and word wi in the corpus may be equal to the probability Pr (wi) of a sentence containingdomain entity de in the corpus. Then the value of PMI would be only determined by Pr (de), and thus many words get thesame PMI value. In a larger corpus, the effectiveness of PMI would gradually emerge (see the PMI values on D4 in Table 8).But acquiring a large domain review corpus is a very high demand for some practical applications. As a result it is not suit-able for feature extraction based on a small corpus.

By combining PMI and TF–IDF, PMI–TFIDF can solve this problem effectively because TF–IDF adds the importance mea-sure of a word in the corpus. For two different words wi and wj, although Pr (de, wi) may be equal to Pr (de, wj), and Pr (wi)

Table 7Performance comparison of measuring the association between feature and domain entity (camera and phone) by PMI and PMI–TFIDF.

Camera (D1) Phone (D3)

PMI PMI–TFIDF PMI PMI–TFIDF

Features Values Features Values Features Values Features Values

camera 13.57 camera 15.23 communications 12.95 phone 19.78environs 13.57 g3 2.15 downloads 12.95 nokia 1.98touch 13.57 canon 1.98 hour 12.95 features 1.89tool 13.57 digital 1.93 station 12.95 light 1.33first-class 13.57 time 1.41 chance 12.95 phones 0.89consumers 13.57 pros 1.23 offers 12.95 speakerphone 0.73grip 13.57 iso 1.11 European 12.95 mobile 0.66year 13.57 quality 1.03 jobs 12.95 t-mobile 0.64iso 13.57 shoot 1.02 outweighs 12.95 quality 0.61automode 13.57 lens 1.00 browser 12.95 sound 0.56media 13.57 features 0.78 capabilities 12.95 voice 0.55today 13.57 pictures 0.77 ears 12.95 motorola 0.53buyer 13.57 point 0.73 night 12.95 talking 0.52skill 13.57 price 0.67 year 12.95 hiss 0.51advice 13.57 noise 0.64 south 12.95 dialing 0.50selects 13.57 nikon 0.54 sounds 12.95 battery 0.45

Table 8Performance comparison of measuring the association between feature and domain entity (mp3 and router) by PMI and PMI–TFIDF.

Mp3 (D4) Router (D6)

PMI PMI–TFIDF PMI PMI–TFIDF

Features Values Features Values Features Values Features Values

mp3 35.78 player 5.07 opinions 10.86 router 4.27generation 35.78 mp3 3.82 router 10.86 table 1.19mini 35.78 software 1.08 construction 10.86 tool 0.47m3u 35.78 ipod 1.05 adjustment 10.86 plunge 0.38weekend 35.78 quality 0.44 3HP+ 10.86 speed 0.37equilizer 35.78 memory 0.42 Internet 10.86 M12V 0.34gb 35.78 tags 0.36 Cable 10.86 bits 0.30jukebox2 35.78 music 0.35 industry 10.86 Hitachi 0.23artists 23.85 sound 0.32 stores 10.86 price 0.23players 21.14 files 0.27 issue 10.86 control 0.18memory 20.44 computer 0.26 introduction 10.86 problem 0.15drives 17.89 battery 0.24 thing 10.86 time 0.15digital 17.89 gb 0.23 features 10.86 adjustment 0.14encode 17.89 product 0.22 mid-size 10.86 action 0.12winamp 17.89 cd 0.21 performer 10.86 machine 0.12media 11.93 rip 0.18 PC 10.86 working 0.11

C. Quan, F. Ren / Information Sciences 272 (2014) 16–28 25

may be equal to Pr (wj), they could also be distinguished very well because the importance measures ofP

kTFIDF(wi, dk) andPkTFIDF(wj, dk) would be different even in a small corpus.

5.3. Experiments on feature-oriented opinion determination

The experiments on feature-oriented opinion determination were conducted for the four product classes: digital camera(D1, D2, D5), cell phone (D3, D10); mp3 player (D4, D7, D9), and router (D6, D8). Polarities of features have been labeled inthe dataset with ‘‘+’’ (positive) and ‘‘-’’ (negative). The performances are measured using the standard evaluation measures ofprecision (p), recall (r) and F-score (f), f = 2pr/(p + r). We use the features extracted by the previous stage for evaluating fea-ture-oriented opinion determination.

For effectiveness evaluation on feature-opinion pair extraction and feature-oriented opinion lexicon generation, we com-pare the results with different experimental setups.

Main experiment steps:

Step 1: Feature-opinion pair extraction based on the dependency distance. We use the general sentiment lexicon GI toestimate parameter b. We collect the sentences containing only one f (f e F) and only one o (o e GI) in DRC, and then bis set by the mean value of dd(f, o). In this experiment, b = 7;

Table 9The comparison of feature-oriented opinion determination using different experimental setups (E1, E2, and E3).

Review domain Parameter E1 E2 E3

k p r f p r f p r f

Digital camera 5.0 0.62 0.70 0.66 0.81 0.62 0.70 0.85 0.67 0.75Cell phone 2.5 0.63 0.72 0.67 0.85 0.64 0.73 0.87 0.69 0.77Mp3 player 3.0 0.64 0.75 0.69 0.83 0.67 0.74 0.86 0.68 0.76Router 0.5 0.64 0.70 0.67 0.82 0.63 0.71 0.85 0.70 0.77

Avg. 0.63 0.72 0.67 0.83 0.64 0.72 0.86 0.69 0.76

26 C. Quan, F. Ren / Information Sciences 272 (2014) 16–28

Step 2: Candidate opinion words filtration for fi based on the value of PMI–TFIDF(fi, oj) (The threshold value k is empiricallyset for different domains, see Table 9.);Step 3: SVM based on Polynomial kernel (PK) function for feature-oriented opinion determination using the general sen-timent lexicon (GI). The parameters of PK (Eq. (16)) are set by c = 0, d = 0.5;Step 4: SVM based on Polynomial kernel (PK) function for feature-oriented opinion determination using generated fea-ture-oriented opinion lexicons.Experimental setups:E1: Step 1 + Step 3;E2: Step 1 + Step 2 + Step 3;E3: Step 1 + Step 2 + Step 4;

Table 9 compares the results with the above different experimental setups.The comparison of E1 and E2 shows the effectiveness of candidate opinion words filtration by computing PMI–TFIDF(fi, oj).

As shown in Table 9, the results obtained by adding candidate opinion words filtration have 5% improvement of F-score overnot adding candidate opinion words filtration, which confirms the usefulness of PMI–TFIDF for opinion words filtration. E2also demonstrates that PMI–TFIDF is a powerful measure for evaluating the association of related words.

The comparison of E2 and E3 shows the effectiveness of using generated feature-oriented opinion lexicons by our method.Table 9 shows that the results obtained by using the generated feature-oriented opinion lexicons have much higher recall,precision, and F-score than using GI lexicons, about 3%, 5% and 4% higher respectively. The results indicate that feature-ori-ented opinion lexicons are helpful for feature-oriented opinion determination. From E3, we also notice that the method isable to achieve high precision but relatively low recall for this task. The main reason is that some features were not recog-nized correctly in the previous stage of domain feature extraction. Therefore, identifying product features from reviews is thefundamental step in the problem of feature-level SA.

6. Conclusions and future work

In this study, we proposed a method of unsupervised product feature extraction for feature-oriented opinion determina-tion. It applied domain-specificity of words as a form of domain-knowledge. For each domain review corpus, we used a do-main entity (DE) term to represent it. The association between a feature and a domain is embodied by the similarity betweenthe feature and the DE term. The results showed that our approach outperformed other state-of-the-art methods in this task.We also proposed the novel term similarity measure (PMI–TFIDF) to evaluate the association of candidate features and do-main words. Compared with PMI measure, PMI–TFIDF showed better distinction ability.

After that, we further explored the problem of feature-oriented opinion determination. In this problem, we proposed fea-ture-opinion pair extraction based on dependency distance and applied semantic association measure for opinion words fil-tration. Feature-oriented opinion lexicons are generated by combining general sentiment lexicon and domain dependentcorpora. The comparison experiments have indicated that feature-oriented opinion lexicons are better than general senti-ment lexicons for feature-oriented opinion determination.

In the future, we plan to focus on improving the method for feature-oriented opinion lexicons generation. We will thenapply this method for some applications such as product ontology construction [22] and product attribute analysis [23].

Acknowledgments

This research has been partially supported by the National Program on Key Basic Research Project of China (973 Program)under Grant No. 2014CB347600, National Natural Science Foundation of China under Grant No. 61203312, National High-Tech Research & Development Program of China 863 Program under Grant No. 2012AA011103, the Scientific Research Foun-dation for the Returned Overseas Chinese Scholars, State Education Ministry, and Key Science and Technology Program ofAnhui Province under Grant No. 1206c0805039.

C. Quan, F. Ren / Information Sciences 272 (2014) 16–28 27

References

[1] A.R. Backes, D. Casanova, O.M. Bruno, Texture analysis and classification: a complex network-based approach, Inf. Sci. 219 (10) (2013) 168–180.[2] D. Bollegala, N. Okazaki, M. Ishizuka, A preference learning approach to sentence ordering for multi-document summarization, Inf. Sci. 217 (25) (2012)

78–95.[3] G. Bouma, Normalized (Pointwise) mutual information in collocation extraction. from form to meaning: processing texts automatically, in: Proc. of the

Biennial GSCL Conference, 2009, pp. 31–40.[4] S. Brody, N. Elhadad, An Unsupervised aspect-sentiment model for online reviews, in: Proc. of the Joint Human Language Technology/North American

Chapter of the ACL Conference (HLT-NAACL), 2010, pp. 804–812.[5] L. Chen, L. Qi, F. Wang, Comparison of feature-level learning methods for mining online consumer reviews, Expert Syst. Appl. 39 (2012) 9588–9601.[6] I. Chetviorkin, N. Loukachevitch, Extraction of domain-specific opinion words for similar domains, in: Proc. of the Workshop on Information Extraction

and Knowledge Acquisition, 2011, pp. 7–12.[7] F.L. Cruz, J.A. Troyano, F. Enríquez, F.J. Ortega, Knowledge-rich approach to feature-based opinion extraction from product reviews, in: Proc. of the 2nd

International Workshop on Search and Mining User-Generated Contents, 2010, pp. 13–20.[8] S. Das, M. Chen, Yahoo! for Amazon: extracting market sentiment from stock message boards, in: Proc. of the 8th Asia Pacific Finance Association

Annual Conference, 2001, pp. 37–56.[9] X. Ding, B. Liu, The utility of linguistic rules in opinion mining, in: Proc. of the 30th Annual International ACM SIGIR Conference on Research and

Development in Information Retrieval, 2007, pp. 811–812.[10] X. Ding, B. Liu, P. S. Yu, A holistic lexicon based approach to opinion mining, in: Proc. of the International Conference on Web Search and Web Data

Mining (WSDM’08), 2008, pp. 231–239.[11] Z. Dong, Q. Dong, HowNet, 1999. <http://www.keenage.com>[12] A. Esuli, F. Sebastiani, SentiWordNet: a publicly available lexical resource for opinion mining, in: Proc. of the Fifth International Conference on

Language Resources and Evaluation (LREC 2006), 2006, pp. 417–422.[13] V. Giouli, A. Konstandinidis, E. Desypri, X. Papageorgiou, Multi-domain, Multi-lingual named entity recognition: revisiting and grounding the resources

issue, in: Proc. of the Fifth International Conference on Language Resources and Evaluation (LREC 2006), 2006, pp. 59–64.[14] V. Hatzivassiloglou, K. R. McKeown, Predicting the semantic orientation of adjectives, in: Proc. of the 40th Annual Meeting on Association for

Computational Linguistics (ACL 1997), 1997, pp. 174–181.[15] V. Hatzivassiloglou, J. Wiebe, Effects of adjective orientation and gradability on sentence subjectivity, in: Proc. of the International Conference on

Computational Linguistics (COLING 2000), 2000, pp. 299–305.[16] P. Hossein Zadeh, M. Reformat, Assessment of semantic similarity of concepts defined in ontology, Inf. Sci. 250 (25) (2013) 21–39.[17] M. Hu, B. Liu, Mining opinion features in customer reviews, in: Proc. of the Nineteenth National Conference on Artificial Intelligence (AAAI), 2004, pp.

755–760.[18] J. Jiang, C. Zhai, Exploiting domain structure for named entity recognition, in: Proc. of the main conference on Human Language Technology Conference

of the North American Chapter of the Association of, Computational Linguistics, 2004, pp. 74–81.[19] W. Jin, H. Ho, R. Srihari, OpinionMiner. A novel machine learning system for web opinion mining and extraction, in: Proc. of the 15th ACM SIGKDD

International Conference on Knowledge Discovery and Data Mining, 2009, pp. 1195–1204.[20] J. Kamps, M. Marx, R. Mokken, MD. Rijke, Using WordNet to measure semantic orientations of adjectives, in: Proc. of the Fifth International Conference

on Language Resources and Evaluation (LREC 2004), vol. IV, 2004, pp. 1115–1118.[21] L.W. Ku, Y.T. Liang, H.H. Chen, Tagging heterogeneous evaluation corpora for opinionated tasks, in: Proc. of Conference on Language Resources and

Evaluation (LREC 2006), 2006, pp. 667–670.[22] R.Y.K. Lau, C.C.L. Lai, J. Ma, Y.F. Li, Automatic domain ontology extraction for context-sensitive opinion mining, in: Proc. of the International Conference

on Information Systems celebrates (ICIS 2009), 2009, pp. 35–53.[23] G. Li, M. Wang, Z. Lu, R. Hong, Tat-Seng Chua, In-video product annotation with Web information mining, ACM Trans. Multimedia Comput. Commun.

Appl. 8 (4) (2012) 55:1–55:19.[24] B. Liu, Sentiment Analysis and Opinion Mining, Morgan & Claypool Publishers, 2012.[25] T. Ma, X. Wan, Opinion target extraction in Chinese news comments, in: Proc. of the International Conference on Computational Linguistics (COLING

2010), 2010, pp. 782–790.[26] Q. Mei, X. Ling, M. Wondra, H. Su, C. Zhai, Topic sentiment mixture: modeling facets and opinions in weblogs, in: Proc. of the World Wide Web

Conference (WWW’07), 2007, pp. 171–180.[27] B. Mirkina, S. Nascimentoc, Additive spectral method for fuzzy cluster analysis of similarity data including community structure and affinity matrices,

Inf. Sci. 183 (1) (2012) 16–34.[28] D. Nadeau, S. Sekine, A survey of named entity recognition and classification, Linguist. Invest. 30 (1) (2007) 326.[29] B. Pang, L. Lee, S. Vaithyanathan, Thumbs up? Sentiment classification using machine learning techniques, in: Proc. of the 40th Annual Meeting on

Association for Computational Linguistics (ACL 2002), 2002, pp. 79–86.[30] B. Pang, L. Lee, Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales, in: Proc. of the 43th Annual

Meeting on Association for Computational Linguistics ACL’05, 2005, pp. 115–124.[31] A. Popescu, O. Etzioni, Extracting product features and opinions from reviews, in: Proc. of the Conference on Human Language Technology and

Empirical Methods in Natural Language Processing (HLT-EMNLP), 2005, pp. 339–346.[32] D. Putthividhya, J. Hu, Bootstrapped named entity recognition for product attribute extraction, in: Proc. of the Conference on Empirical Methods in

Natural Language Processing (EMNLP), 2011, pp. 1557–1567.[33] G. Qiu, B. Liu, J. Bu, C. Chen, Opinion word expansion and target extraction through double propagation, Comput. Linguist. 37 (1) (2011) 9–27.[34] G. Qiu, B. Liu, J. Bu, C. Chen, Expanding domain sentiment lexicon through double propagation, in: Proc. of the 21st International Joint Conference on

Artificial Intelligence (IJCAI-09), 2009, pp. 11–17.[35] C. Quan, F. Ren, T. He, Sentimental classification based on kernel methods and domain semantic orientation dictionaries, Int. J. Innovat. Comput. Inf.

Control 6 (6) (2010) 2681–2690.[36] F. Ren, M.G. Sohrab, Class-indexing-based term weighting for automatic text classification, Inf. Sci. 236 (1) (2013) 109–125.[37] G. Salton, C. Buckley, Term-weighting approaches in automatic text retrieval, Inf. Process. Manage. 24 (5) (1988) 513–523.[38] C. Scaffidi, K. Bierhoff, E. Chang, M. Felker, H. Ng, C. Jin, Red Opal: product-feature scoring from reviews, in: Proc. of the ACM Conference on Electronic

commerce, 2007, pp. 182–191.[39] B. Snyder, R. Barzilay, Multiple aspect ranking using the good grief algorithm, in: Proc. of the Joint Human Language Technology/North American

Chapter of the ACL Conference (HLT-NAACL), 2007, pp. 300–307.[40] S. Somasundaran, J. Wiebe, Recognizing stances in online debates, in: Proc. of the Joint conference of the 47th Annual Meeting of the Association for

Computational Linguistics (ACL 2009), 2009, pp. 226–234.[41] P.J. Stone, D.C. Dunphy, M.S. Smith, D.M. Ogilvie, The General Inquirer: A Computer Approach to Content Analysis, MIT Press, 1966.[42] C. Strapparava, A. Valitutti, Wordnet-affect: an affective extension of wordnet, in: Proc. of the Forth International Conference on Language Resources

and Evaluation (LREC 2004), 2004, pp. 1083–1086.[43] C. Strapparava, A. Valitutti, O. Stock, The affective weight of lexicon, in: Proc. of the Fifth International Conference on Language Resources and

Evaluation (LREC 2006), 2006, pp. 423–426.

28 C. Quan, F. Ren / Information Sciences 272 (2014) 16–28

[44] A. Tapia-Roseroa, A. Bronselaerb, G. De Tréb, A method based on shape-similarity for detecting similar opinions in group decision-making, Inf. Sci. 258(2014) 291–311.

[45] P. Turney, M. Littman, Measuring praise and criticism: inference of semantic orientation from association, ACM Trans. Inf. Syst. 21 (4) (2003) 315–346.[46] P. Turney, Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews, in: Proc. of the 40th Annual Meeting on

Association for Computational Linguistics (ACL 2002), 2002, pp. 417–424.[47] S. Veselin, C. Cardie, Topic identification for fine-grained opinion analysis, in: Proc. of the International Conference on Computational Linguistics

(COLING 2008), 2008, pp. 817–824.[48] M. Wang, X. Hua, J. Tang, R. Hong, Beyond distance measurement: constructing neighborhood similarity for video annotation, IEEE Trans. Multimedia

11 (3) (2009) 465–476.[49] M. Wang, G. Li, Z. Lu, Y. Gao, T. Chua, When Amazon Meets Google: product visualization by exploring multiple information sources, ACM Trans. Int.

Technol. 12 (4) (2013) (Article 12).[50] J. Wiebe, T. Wilson, R. Bruce, M. Bell, M. Martin, Learning subjective language, Comput. Linguist. 30 (3) (2004) 277–308.[51] J. Yu, Z. Zha, M. Wang, T. Chua, Aspect ranking: eliciting important product aspects from online consumer reviews, in: Proc. of the Annual Meeting on

Association for Computational Linguistics (ACL), 2011.[52] Z. Zhang, J. Iria, Improving domain-specific entity recognition with automatic term recognition and feature extraction, in: Proc. of the Seventh

International Conference on Language Resources and Evaluation (LREC 2010), 2010, pp. 19–21.[53] L. Zhang, B. Liu, Extracting and ranking product features in opinion documents, in: Proc. of the International Conference on Computational Linguistics

(COLING 2010), 2010, pp. 1462–1470.