analysis of regional information collected from twitter · the web mining is a kind of data mining....

IJBSCHS Biomedical Soft Computing and Human Sciences, Vol.x,No.x,pp.000-000) [Original article] Copyright©1995 Biomedical Fuzzy Systems Association

(Accepted on 20xx.xx.xx)

1

Analysis of Regional Information Collected from Twitter

Yoshiyuki MATSUMOTO and Yoshiyuki YABUUCHI

Faculty of Economics, Shimonoseki City University

Abstract: Recently, people are collecting regional information using the Internet. Also, the SNS and blog are important tools to transfer information such as regional information and internet user's experience. The development of information and communication technology has enabled to dissem-inate information easily. Therefore, to write personal experiences on “word of mouth” has a signif-icant public influence. In this study, we collect local information from the big data on the Internet. We focused on Twitter that can transmit information easily. Our objective is to collect regional in-formation from Twitter, and to acquire knowledge information by analyzing. Keywords Twitter, Web mining, Regional information.

1.Introduction

Recently, consumers are collecting regional infor-mation and tourist information from the Web page. In the past, consumers had to collect information from the trav-el agency, magazines and TV. Consumers have enabled the information collected using the Web by a computer and smart phones have become popular. In addition, tourists post information about the experiences in a Blog or SNS. Information technology has evolved; it has be-come possible to transmit the information easily in com-parison with the past. Then, describing the personal ex-perience “review” information has become important. It is necessary to perform the information transmission and information gathering on the Web for tourism promotion and regional development. On the other hand, there is an enormous amount of information on the Web. Method to find the necessary information from the Web has been explored. Knowledge acquisition is studied from auto-matically collected and stored data on the Internet. Stud-ies to find specific patterns from the text information of the Web page has been actively conducted [1] [2].

In this research, it is an object of collection and feature extraction area information of “Shimonoseki” from vast amounts of data on the Web. We think to be able to ac-quire knowledge by collecting and analyzing information transmitted from various media on the Internet. As in-formation transmission means on the Internet, there is such as traditional Web page, Blog and SNS. SNS is dis-semination of information in individual easy and high immediacy. SNS is an advantage interactivity compared with other media. In this research, we focused on the Twitter in the SNS. Twitter is high immediacy and easy to disseminate information available. We collect the in-formation that originates from Twitter, and aim to acquire knowledge from the information.

2. Collecting information from Twitter

The SNS is a community type of website to promote the communication between humans. In addi-tion, SNS is a push-type information service. Message is transmitted from the automatic setting target SNS. Information transmission means of “Social Button” exists in the SNS. SNS is Web page high information diffusion resistance and interactivity.

SNS users is 4,965 million people in the Internet users 9,556 million people of Japan (Fig. 1) according to the survey on the SNS use trends in 2013 by ITC Research Institute[4]. In addition, according to the

2-1-1 Daigaku-cho, Shimonoseki, Yamaguchi 751-8510 Japan Phone: +81-83-252-0288 e-mail: [email protected]

15

IJBSCHS

［Original Article］Biomedical Soft Computing and Human Sciences, Vol.20, No.2, pp.15-22

CopyrightⒸ1995 Biomedical Fuzzy Systems Association(Accepted on 2015.11.30)

Biomedical Soft Computing and Human Sciences, Vol.x, No.x, (20xx)

2

result that the company was investigated in 2013, Fa-cebook users 34%, Twitter users 26%, mixi user is 22% (Fig. 2). The three of SNS has been utilized very well in Japan. Mixi is Japan's largest SNS website. LINE is the most prevalent free message exchange and call application in Japan.

In this Research, we will collect and analyze infor-mation from Twitter. When collecting tourist infor-mation from the Internet, the user is considered to emphasize Reviews. If coherent information is needed, towards the collection of information from travel magazines and travel agency which is a conventional media are suitable. Twitter is highly responsive in the SNS, and can be easily dissemination and discover Reviews. We think it is the media that Twitter is the most suitable in order to obtain a real-time traveler information.

In this Research, we are using the TwitterAPI that Twitter-supplied. TwitterAPI is can post and search tweets from client applications and Web pages. This TwitterAPI has been changed from version 1.0 to ver-sion 1.1 from June 12, 2013 [4]. This change is aimed at elimination of malicious applications and bot. Therefore, it is necessary authentication procedure about the request was not necessary until now. Also, API call limit the number has been cut. OAuth authen-tication is used for the authentication procedure. The OAuth authentication is an authentication method that enables delivery of data between services and applica-tions without passing the User ID and Password. It is necessary to previously register the development ap-plication to OAuth authentication in Twitter. It sets the application URL and allowed the operation for regis-tration. When you register, it is possible to get the Ac-cess token and Access token secret from Twitter. If the application using the obtained Access token and Ac-cess token secret by OAuth authentication, it is possi-ble to use the TwitterAPI. It uses the GET search / tweets in the TwiiterAPI to search from Twitter. As a result, it will be able to any search from application to Twitter.

Fig. 1. Popular situation of SNS

Fig. 2. Usage rate of SNS and Call application

3. Text mining

The Web mining is a kind of data mining. Data mining is one of the data analysis methods, and intended to search for useful information from the database or data warehouse. Data mining is an analytical approach to the data that has been trimmed to on the database and data warehouse. However, web mining is analyzed the data on the Internet. Data stored in the database with data that has been adjusted formally. It is relatively easy to analyze. Data on the Web is mostly text data. It has not been ad-justed formally. It is necessary to use a technique called text mining for analyzing text data.

Text mining is a data analysis technique that targets the sentence such as novels, newspapers and Web page. Text mining’s purpose is extraction of specific themes from a large amount of text data, and document classi-fication[5]. It is possible to analyze without using a computer if the text data is a small amount. However, an essential technology when analyzing a large amount of data that is stored on the Internet. To dis-

16

Biomedical Soft Computing and Human Sciences, Vol.20, No.2, 2016

N. SHIRAHAMA and T. YANARU: An Approach to Construct an Emotional Dialogue System

3

cover useful blogger originating the information is high immediate and importance by analyzing the blog information on the Internet [6], and visualization extracts the Web site and search keywords that the user has impressed on the Internet [7] has been re-search.

4. Collection of regional and tourist in-formation

In the past, it was not easy to access regional in-

formation and tourist information. In recent years, it be-gan to various reviews and articles are written in such blog and SNS by the development of the Internet services. The purpose of this research is to collect regional and tourist information from the Web. And, Twitter is a source of the information. Twitter messages one is within 140 characters. Twitter data collection is efficient com-pared to the blog. Therefore, we were collected regional and tourist information from the writing of Twitter. Text mining target is Japanese. We have collected the Japa-nese tweets. We analyzed the Twitter data using R.

We have analyzed two data as follow. Long-term data: July 9, 2013 to April 15, 2014 Short-term data: March 5 14:00, 2014 to March 6 14:00

These are the Tweets about Shimonoseki. Long-term data is 5,646 items. It does not include the retweets. Short-term data is 1,478 items. It includes the retweets. We find the interest of Internet users in the long-term data. And it aims to determine the amount of interest in the short-term data. Therefore, in this research we take the analysis procedure as following.

Long-term data Purpose: it examines the user’s interest. Step 1: We will confirm the co-occurrence of three terms. We find the three terms that have been written on the Twitter. Step 2: We will check the connection of the term.

Short-term data Purpose: it confirms the concerns found in the long-term data. Step 1 and Step 2 are the same as above.

It contains symbols not related to the meaning of a sentence in the text. Therefore, we will morphological analysis and parsing after cleaning the data in a normal text mining. However, tweet contains many hash tags and emoticons. It is inefficient when cleaning for tweet at first. Therefore, it is cleaning after the morphological

analysis and parsing. It is called a morpheme unit of minimum character string having a meaning in a lan-guage. Work for adding part-of-speech information to divide the sentence into words is called a morphological analysis. If we replace the morpheme to the terms, text is the connection of the term. It is possible to analyze the combination and number of occurrences of the term.

Method of aggregate the n-number of term connection is called the n-gram. 2 number of terms are called bigram, 3 number of terms are called trigram. In this research, we examine the co-occurrence of 3-terms and 2- terms. It examines the roughly co-occurrence in the trigram, and statistical processing in order to check the connection with bigram.

Among the indicators of the n-gram, we used the t value and MI (Mutual Information) values. MI value is written the use of probability p(x), p(y) for random vari-able x, y as follows.

)()(),(logYPXP

YXPMI (1)

In this Research, we have determined a significant difference of co-occurrence by using the t value and MI value. (1) Analysis of long-term data

The combination of the tri-gram in the long-term data was 129,870. These are included many hash tags and emoticons. Number of combinations when the re-moval of these are 3,320, if limited to the co-occurrence of 3 or more. Network diagram shows the term in the node, the context of co-occurrence in the direction, a little and more of the co-occurrence in the thickness of the line. The network is created from the adjacency ma-trix of the term. The adjacency matrix shows the connec-tion by {0, 1}. Fig. 3 is a graph using only the combina-tion of the co-occurrence number 50 or more. It was dif-ficult to understand the graph by using the co-occurrence number 50 or less. Information that can be read from Fig. 3 is a Shimonoseki, Kaikyo, Tsunoshima bridge, and Research whaling. It will check other information bigram considering as above. The search in the bigram, we ex-amine the before and after five morpheme is to determine the morpheme node. However, it morpheme the node is set manually. The data used in the analysis is the result of searching by keyword Shimonoseki. Therefore, Fig. 3 has shown that it significant information is Shimonoseki. We examined the bigram the initial node as Shimonoseki. The bubble chart is shown in Fig. 4. Bubble radius indi-

17

Y. MATSUMOTO and Y. YABUUCHI: Analysis of Regional Information Collected from Twitter


4

cates the relative number of co-occurrence. The t value and MI value shown in Table 1. We investigated the Treaty, Bridge, Shop, Seamall, Puffer, Aquarium, Strait, Arukaport, Station, Independent, Harbor, Departure, Walk, Born and Intermediate of 15 terms in bigram.

Fig. 3. Network diagram of long-term data (co-occurrence of 50 or more)

Fig. 4. Bubble chart by node for Shimonoseki (co-occurrence of 50 or more)

Table 1. T and MI values of bigram by node for Shi-monoseki (top-ranked 22)

Label t MI 001 City 36.612 2.361002 Yamaguchi 26.651 1.735003 Prefecture 24.450 1.855004 Station 16.926 2.195005 Town 13.543 1.712006 Treaty 11.704 2.061 007 Harbor 11.275 2.316 008 Houhoku 9.751 1.672 009 Kanda 9.614 2.438 010 Seamall 9.594 2.448 011 Departure 9.513 2.330 012 Bridge 9.213 1.701 013 Municipal 8.813 2.355 014 Shop 8.294 1.652 015 JR 7.876 1.801 016 Redline 6.811 2.428 017 First 6.802 1.782 018 IC 6.153 2.078 019 Sanyo Line 6.122 1.927 020 Aquarium 6.025 1.739 021 Walk 5.853 2.537 022 Graduate 5.847 1.893

Fig. 5 shows the bubble chart with the treaty as nodes.

Table 2 shows the t value and MI value. This bigram shows that it has entered into a Sino - Japanese Peace (Shimonoseki) Treaty.

Similarly, we have created a Fig. 6 and Table 3 the Bridge as a node. We examined the writing to Twitter meaningful at the local and tourist information using the t value and MI value in the bigram. It was found that there was a write Tsunoshima Bridge, Sino - Japanese Peace Treaty, History education, Scientific whaling, Kaikyokan, Puffer, Regional information, Event information. The Period of long-term data is from July to April. Tsunoshi-ma Bridge, Aquarium and Fireworks are hot topics dur-ing the warmer months. Puffer fish and its related matters are hot topics in the winter. The long-term data, it has confirmed that written on Twitter. However, it was not possible to find the regional and tourist information other than these. The reasons are as follows. Most of Twitter users are in their 20s and 30s. Most of these age groups is sightseeing explore the urban areas, and it is less likely to

18



5

sightseeing places like Shimonoseki. Age group to sight-seeing local city is the older than Twitter users. Therefore, we think that we have not obtained a distinctive regional and tourist information in this analysis.

Fig. 5. Bubble chart by node for treaty (co-occurrence of 5 or more)

Table 2. T and MI values of bigram by node for treaty (top-ranked 22)

Label t MI 001 Shimonoseki 11.841 2.115 002 Qing 10.168 5.879 003 War 9.422 6.349 004 Year 7.617 3.591 005 After 7.140 5.702 006 Peace 7.089 7.079 007 Independence 6.759 4.858 008 Qing or 6.355 7.051 009 Education 6.334 6.526 010 Teach 6.311 6.123 011 History 6.212 5.068 012 Korea 6.132 4.214 013 Nichi (Japan) 5.821 1.907 014 Yamaguchi -5.785 -1.843 015 Japan 5.620 3.717 016 No. 5.553 4.025 017 Entered into 4.955 6.788 018 Qing Dynasty 4.722 6.020 019 Pact 4.652 6.925 020 One 4.600 4.028 021 Representative 4.571 5.301 022 Tie 4.346 5.144

Fig. 6. Bubble chart by node for Bridge

Table 3. T and MI values of bigram by node for Bridge

Label t MI 001 Prefecture 15.958 4.486 002 Yamaguchi 15.895 4.057 003 Tsunoshima 12.341 6.050 004 Shining 11.575 7.060 005 Take a bridge 11.492 6.097 006 Sea 11.410 5.123 007 City 10.906 3.231 008 Shimonoseki 9.364 1.756 009 Houhoku 8.006 4.786 010 is -6.577 -1.979 011 Kanmon 1.822 2.434 012 Nichi (Japan) -4.121 -1.423

(2) Analysis of Short-term data

The combination of the tri-gram in the short-term data was 17,150. These are included many hash tags and emoticons. Number of combinations when the removal of these are 460, if limited to the co-occurrence of 3 or more.

Only main results are shown as follow. Fig. 7 is a graph using only the combination of the co-occurrence number 20 or more. Information that we found from the

19



6

tri-gram are Events, Tiger puffer and Tsunoshima Bridge. Tsunoshima Bridge was also found from the long-term data. However, Tiger puffer and Events not found from it.

We used a bigram order to confirm as above. Search interval is a five morpheme before and after the node same as the long-term data. Initial node is Shimonose-ki (Fig. 8).

Term of bigram was shown in Table 4. It is con-spicuous terms such as Station, Events and Idol of Shimonoseki originating to be used everyday. It ana-lyzed for 13 terms (Holding, Advance sale, Store, Opening, Tiger, Treaty, AmiPara, Direct delivery, Domestic, Appeal, Seamall, Seafood, Hope) shown in Table 4.

We have set the node to the Tiger, Direct delivery and Domestic. In that case, it was connected to the keyword Habitat, Shimonoseki, Domestic, Tiger puff-er set and Direct delivery. It shows a Fig. 9 and Table 5 that specified the Tiger to the node.

In addition to the above, “Sino-Japanese Peace Treaty” and “Shimonoseki local Idol PalmSugar star-ring in Shimonoseki opening event” was found. Write frequency of many users of the post had been noticea-ble.

We could not find other than Tiger puffer set and Sino-Japanese Peace Treaty with respect to regional and tourist information. It was a new discovery Shi-monoseki departure idle attention has been paid.

In summary, Tiger puffer set , Shimonoseki Station opening event , Sino-Japanese Peace Treaty and Store information are topics on Twitter from short-term data. Similar to the analysis in the long-term data, Enter-tainer, Event and Store information are often in writ-ing content is no wonder in consideration of the Twit-ter user age group. If you invite a Twitter user to tour-ist destinations, we think those keywords are effective. The following do not use Twitter but it is a youth-facing events. It has been held every year the music festival summoned the lock hip-hop reggae art-ists in Awaji Island. It has held the events several times a year to work with animation production com-pany in Tokushima city. When planning an event in which the young people to target in Shimonoseki, it is considered as being inspired by the analysis of this study.

Fig. 7. Network diagram of Short-term data (co-occurrence of 20 or more)

Table 4. T and MI values of bigram by node for Shi-monoseki

Label t MI 001 City 14.280 1.935002 Station 8.407 1.814003 Hakata 7.705 1.879004 Ueno 7.613 1.940005 Holding 7.476 1.887006 Kanda 7.318 1.939007 Advance sale 7.038 1.954008 Hope 5.011 1.894009 Opening 4.808 1.954010 Tax accountant 4.535 2.539011 Store 4.370 1.618012 PalmSuger 4.063 1.954013 Office 3.895 2.954014 Hanano 3.895 2.954015 Tiger 3.875 1.773016 Treaty 3.732 1.899017 AmiPara 3.709 1.954018 JUNCHI 3.634 1.954019 Waves 3.558 1.954020 Detail 3.557 1.664021 Direct delivery 3.504 1.892022 Pin 3.480 1.954

20



7

Fig. 8. Bubble chart by node for Shimonoseki (co-occurrence of 20 or more)

Fig. 9. Bubble chart by node for Tiger

Table 5. T and MI values of bigram by node for Tiger

Label t MI 001 Puffer 2.973 0.987002 Direct delivery 3.714 2.148003 Set 3.308 1.689004 Domestic 3.676 2.209005 Shimonoseki -61.925 -3.667006 Habitat 1.910 1.847007 King 1.753 2.209

5.Conclusion In this Research, we collected regional and tourist information from the Internet. The collected data was analyzed. We have examined whether they are useful to tourism promotion and regional devel-opment. We have analyzed the tweed of Twitter is a SNS on the Internet. Twitter is an excellent SNS in bulletins and diffusion. Therefore, we considered it is possible to obtain various information from Twitter. We analyzed the two types data. These data are long-term data (July 2013 to July 2014) and short-term data (March 5, 2014).

Long-term data contained many warm season of tour-ist information such as Tsunoshima Bridge, Kaikyokan and Fireworks. Also, we have seen many Tweets about such as Puffer fish and Sino - Japanese Peace (Shimono-seki) Treaty. Short-term data was seen many tweets about the Timing of the event information, Sightseeing infor-mation and Puffer fish. If anything, we were able to get a lot of regional and tourism data from long-term data.

This Web mining, general (It has been posted on the city’s tourism website [8]) tourist destination (Karato, Kamonwharf, Akama Shrine, Ganryu island and Chofu, etc.) has not been hardly detected as a keyword.

We think that these tourist destinations are not very at-tractive in the 20s and 30s that are mainly used the SNS. However, Tsunoshima Bridge & RedLine (Live house) & PalmSugar (Local Idol) was detected many as keywords.

Not only Puffer & Historical that has been said until now, we considered necessary to tourism resources de-velopment by different point of view.

Tourist destinations become hot topic in SNS, we think that spots of youths or photo post easy.

We were able to obtain knowledge of tourism from Twitter. Especially long-term data was contained much tourist information. We think that this knowledge will help to tourism development.

6. Discussion

In this research, it was analyzed mainly tweets con-taining “Shimonoseki”. However, another of knowledge for Web mining is also considered likely to be obtained by other related keywords such as “Kanmon”, “Yama-guchi” and “Moji”. In addition, we think that there is a need for a comparative research of the famous tourist destination (Hokkaido, Okinawa and Kyoto).

21



8

References

[1] Kosuke SHINODA, Takeshi SAKAKI, Fujio TORIUMI, Kazuhiro KAZAMA, Satoshi KURIHARA, Itsuki NODA, Yutaka MATSUO: “How Did We Used Twitter, How Communicate in Twitter at Catastrophic Earthquake in Japan”, Japan Society for Fuzzy Theory and Intelligent Informatics, Vol.25, No.1, pp.598-608, 2013, in Japanese

[2] Yuzu UCHIDA, Kenji ARAKI, Jun YONEYAMA, “A Method of Onomatopoeia Sentences Extraction from Blog Entries”, Japan Society for Fuzzy Theory and Intelligent Informatics, Vol. 24, No. 3 pp.811- 820, 2012, in Japanese

[3]ICT Research & Consulting Inc. : “Research on SNS usage trends in 2013”, http://www.ictr.co.jp/report/20130530000039.html (Reference date: June 30, 2013)

[4] Twitter, Inc., “Changes coming in Version 1.1 of the Twitter API”, https://blog.twitter.com/2012/changes-coming-to-twittwi-api (Reference date: June 30, 2013)

[5] Manabu Nii, Shota Miyake, Kazunobu Takahama, Atsuko Uchinuno, Reiko Sakashita, “Consideration about Utilizing Text Architecture for Making Feature Vectors in Classifying Nursing-Care Texts”, IEEE International Conference on Systems, Man, and Cy-bernetics, pp.1817-1821, 2013

[6] Shinsuke NAKAJIMA, Junichi TATEMURA, Yoshinori HARA, Katsumi TANAKA, Shunsuke UEMURA, “A Method of Blog Thread Analysis to Discover Important Bloggers”, Japan Society for Fuzzy Theory and Intelligent Informatics, Vol.19, No.2, pp.156-166, 2007, in Japanese

[7] Tsuyoshi MURATA, Kota SAITO, “Extraction and Visualization of Web Users’ Interests Using Site-Keyword Graphs”, Japan Society for Fuzzy Theory and Intelligent Informatics, Vol.18, No.5, pp.701-710, 2006, in Japanese

[8]Shimonoseki City, “Shimonoseki tourism website”, http://www.city.shimonoseki.yamaguchi.jp/kanko/modmo.html (Reference date: March 31, 2014)

Yoshiyuki MATSUMOTO

He is a Professor of Faculty of Economics,

Shimonoseki City University, Japan.

He is a member of BMFSA.

Yoshiyuki YABUUCHI

He is a Professor of Faculty of Economics,

Shimonoseki City University, Japan.

He is a member of BMFSA.

Photograph of

the first author

Photograph of

the second author

22


analysis of regional information collected from twitter · the web mining is a kind of data mining....

Documents