identifying twitter spam by utilizing random forests · 1 introduction i top social media platforms...

Identifying Twitter Spam by UtilizingRandom Forests

Humza HaiderDivision of Science and Mathematics

University of Minnesota Morris

2017-04-15

1

Introduction

I Top social media platforms

I 500 million tweets per dayI Attracts spammers and malicious usersI Twitter spam: Any unsolicited, repeated actions that negatively

impact other usersI How can we identify spammers?

I Manual classificationI URL blacklistingI Machine learning classification

Humza Haider | Identifying Twitter Spam by Utilizing Random Forests

1

Introduction

I Top social media platformsI 500 million tweets per day

I Attracts spammers and malicious usersI Twitter spam: Any unsolicited, repeated actions that negatively




1

Introduction

I Top social media platformsI 500 million tweets per dayI Attracts spammers and malicious users

I Twitter spam: Any unsolicited, repeated actions that negativelyimpact other users

I How can we identify spammers?I Manual classificationI URL blacklistingI Machine learning classification


1

Introduction

I Top social media platformsI 500 million tweets per dayI Attracts spammers and malicious usersI Twitter spam: Any unsolicited, repeated actions that negatively

impact other users

I How can we identify spammers?I Manual classificationI URL blacklistingI Machine learning classification


1

Introduction





1

Introduction



I Manual classification

I URL blacklistingI Machine learning classification


1

Introduction



I Manual classificationI URL blacklisting

I Machine learning classification


1

Introduction





2

Outline

BackgroundDecision TreesRandom ForestsModel Evaluation

MethodsTweet and User Content FeaturesGeo-Tagged FeaturesTime Features

Results

Conclusion


3

BackgroundDecision Trees

I Decision Trees

I Machine learning technique for classificationI Classifies an observation based on features available in a dataset


3


I Decision TreesI Machine learning technique for classification

I Classifies an observation based on features available in a dataset


3


I Decision TreesI Machine learning technique for classificationI Classifies an observation based on features available in a dataset


4


URL Account Age Reported ClassNo Old Yes Not SpamNo Old Yes Not SpamNo Old No Not SpamNo New No Not SpamYes New Yes SpamNo New Yes SpamNo Old Yes SpamYes New No Not Spam

......

......


5




6



7




8



9


URL Account Age Reported ClassNo Old Yes Not SpamNo Old Yes Not SpamNo Old No Not SpamNo New No Not Spam

Yes New Yes SpamNo New Yes SpamNo Old Yes SpamYes New No Not Spam


10





11



12





13



14


URL Account Age Reported ClassYes Old Yes TBD


15



16


I How are splits decided?

I EntropyI Information Gain

I Trees seem pretty neat! Why do I need a whole forest?I Disagreement in decisions between different trees


16


I How are splits decided?I Entropy

I Information GainI Trees seem pretty neat! Why do I need a whole forest?

I Disagreement in decisions between different trees


16


I How are splits decided?I EntropyI Information Gain



16



I Trees seem pretty neat! Why do I need a whole forest?

I Disagreement in decisions between different trees


16





17



18

BackgroundRandom Forests

I How do we handle disagreement?

I Train many trees on samples of the data (Bagging)I Don’t let trees access all the features (Feature Bagging)

I After we make a bunch of trees, how do we combine them?I Majority vote


18


I How do we handle disagreement?I Train many trees on samples of the data (Bagging)

I Don’t let trees access all the features (Feature Bagging)I After we make a bunch of trees, how do we combine them?

I Majority vote


18


I How do we handle disagreement?I Train many trees on samples of the data (Bagging)I Don’t let trees access all the features (Feature Bagging)



18



I After we make a bunch of trees, how do we combine them?

I Majority vote


18





19



Source: https://i.ytimg.com/vi/ajTc5y3OqSQ/hqdefault.jpg

20

BackgroundModel Evaluation

I How do we evaluate a random forest’s performance?

True PositiveSpam

Spam

False Positive

Not Spam

False NegativeNot Spam True Negative

Pre

dict

ion

Truth


21


Accuracy

I “How many tweets were correctly identified?”

True PositiveSpam

Spam

False Positive

Not Spam


Pre

dict

ion

Truth

Accuracy

Accuracy =TP + TN

TP + TN + FP + FN


22


Precision (p)

I “How good is our spam prediction?"

True PositiveSpam

Spam

False Positive

Not Spam


Pre

dict

ion

Truth

Precision

Precision =TP

TP + FP


23


Recall (r )

I “How much spam was identified?"

True PositiveSpam

Spam

False Positive

Not Spam


Pre

dict

ion

Truth

Recall

Recall =TP

TP + FN


24


I F-measure (F )I Harmonic mean of Precision and RecallI Equally weights both Precision and Recall

F-Measure

F-measure = 2 · Precision · RecallPrecision + Recall


25

Outline



Results

Conclusion


26

MethodsTweet and User Content Features

I Chen et al. identify tweets, as opposed to users

I Utilized 12 features directly accessible from a tweetI 6 user featuresI 6 tweet features

I User FeaturesI Age in days of accountI Number of followers, followeesI Number of tweets

I Tweet FeaturesI Number of hashtags (#)I Number of mentionsI Number of URLs


26


I Chen et al. identify tweets, as opposed to usersI Utilized 12 features directly accessible from a tweet

I 6 user featuresI 6 tweet features




26



I 6 user features

I 6 tweet featuresI User Features

I Age in days of accountI Number of followers, followeesI Number of tweets



26



I 6 user featuresI 6 tweet features




27


I Two sets of testing data.


28

Outline



Results

Conclusion


29

MethodsGeo-Tagged Features

I So, what is a geo-tagged tweet?


30


I Guo and Chen identify non-personal usersI SpammersI BotsI Business accounts

I Features:I Tweeting Speed


30



I Features:

I Tweeting Speed


30



I Features:I Tweeting Speed


31


I 19.6 Miles


31


I 19.6 MilesI From Clontarf at 7:53 PM


31


I 19.6 MilesI From Clontarf at 7:53 PMI From Morris at 8:00 PM


31


I 19.6 MilesI From Clontarf at 7:53 PMI From Morris at 8:00 PMI Tweeting speed = 19.6 miles

7 minutes = 2.8 miles per minute (168 MPH)


32


I Features:

I Max SpeedI Mean SpeedI Max Distance (connected to Max Speed)I Mean number of times a user exceeds 90 MPH per month

I County based features


32


I Features:I Max Speed

I Mean SpeedI Max Distance (connected to Max Speed)I Mean number of times a user exceeds 90 MPH per month



32


I Features:I Max SpeedI Mean Speed

I Max Distance (connected to Max Speed)I Mean number of times a user exceeds 90 MPH per month



32


I Features:I Max SpeedI Mean SpeedI Max Distance (connected to Max Speed)

I Mean number of times a user exceeds 90 MPH per monthI County based features


32


I Features:I Max SpeedI Mean SpeedI Max Distance (connected to Max Speed)I Mean number of times a user exceeds 90 MPH per month



33

I Number of times a user crosses county borders per monthI Mean number of counties a user has been to per month


33

I Number of times a user crosses county borders per month

I Mean number of counties a user has been to per month


33

I Number of times a user crosses county borders per monthI Mean number of counties a user has been to per month


34

Outline



Results

Conclusion


35

MethodsTime Features

I Washha et al. classify spammers on a user level

I Motivated to use time since altering time dependent features is achallenge.

I Features that spammers can easily manipulate:I Number of URLsI Number of HashtagsI Including Geo-tags


35


I Washha et al. classify spammers on a user levelI Motivated to use time since altering time dependent features is a

challenge.

I Features that spammers can easily manipulate:I Number of URLsI Number of HashtagsI Including Geo-tags


35



challenge.I Features that spammers can easily manipulate:

I Number of URLsI Number of HashtagsI Including Geo-tags


35




I Number of URLs

I Number of HashtagsI Including Geo-tags


35




I Number of URLsI Number of Hashtags

I Including Geo-tags


35




I Number of URLsI Number of HashtagsI Including Geo-tags


36


FeaturesI Differences in Account Age

I Spammers have multiple accountsI Likely to be made at the same timeI FollowersI FolloweesI Bi-directional relationships

I Time weighted correlations:I URLsI MentionsI Hashtags

I Tweet similarity weighted by time


36



I Spammers have multiple accounts

I Likely to be made at the same timeI FollowersI FolloweesI Bi-directional relationships




36



I Spammers have multiple accountsI Likely to be made at the same time

I FollowersI FolloweesI Bi-directional relationships




36



I Spammers have multiple accountsI Likely to be made at the same timeI Followers

I FolloweesI Bi-directional relationships




36



I Spammers have multiple accountsI Likely to be made at the same timeI FollowersI Followees

I Bi-directional relationshipsI Time weighted correlations:

I URLsI MentionsI Hashtags



36







36




I Time weighted correlations:

I URLsI MentionsI Hashtags



36




I Time weighted correlations:I URLs

I MentionsI Hashtags



36




I Time weighted correlations:I URLsI Mentions

I HashtagsI Tweet similarity weighted by time


36







37

Outline



Results

Conclusion


38

Results

I Accuracy:True PositiveSpam

Spam

False Positive

Not Spam


Pre

dict

ion

Truth

TP+TNTP+FP+TN+FN "How many tweets were

correctly identified?"

I Precision (p):True PositiveSpam

Spam

False Positive

Not Spam


Pre

dict

ion

Truth

TPTP+FP "How good is our spam

prediction?"

I Recall (r ):True PositiveSpam

Spam

False Positive

Not Spam


Pre

dict

ion

Truth

TPTP+FN "How much spam was

identified?"I F-measure (F ): 2 · precision·recall

precision+recall

Model Results of the Three Studies

Study %Spam p r F Accuracy

User/Tweet Features: I 50.0% 0.929 0.943 0.936 0.936User/Tweet Features: II 5.0% 0.929 0.407 0.566 0.978Geo-tagged Features 21.4% 0.959 0.959 0.958 0.959

Time Features 46.9% 0.932 0.931 0.931 0.931


39

Conclusion

I Classification via random forestI Recall (r ) may drop when test set contains a low proportion of

spamI Future work: Apply this finding to geo-tagged tweets and time

featuresI Future spam classification by Twitter: Random forests?


40

Acknowledgments

I Peter DolanI For acting as my advisor for this research project and his continued

friendship for the past few yearsI Elena Machkasova

I For the invaluable advice and insightful comments throughout theentire senior seminar course

I Jacob OpdahlI For the exceptional revisions and comments he made as my alumni

review


41

References I

Fabricio Benevenuto, Gabriel Magno, Tiago Rodrigues, andVirgilio Almeida, Detecting spammers on twitter, Collaboration,electronic messaging, anti-abuse and spam conference (CEAS),vol. 6, 2010, p. 12.

Leo Breiman, Random forests, Machine learning 45 (2001),no. 1, 5–32.

Chao Chen, Jun Zhang, Xiao Chen, Yang Xiang, and WanleiZhou, 6 million spam tweets: A large ground truth for timelytwitter spam detection, 2015 IEEE International Conference onCommunications (ICC), IEEE, 2015, pp. 7065–7070.

Diansheng Guo and Chao Chen, Detecting non-personal andspam users on geo-tagged twitter network, Transactions in GIS18 (2014), no. 3, 370–384.


42

References II

Miroslav Kubat, An introduction to machine learning, 1st ed.,Springer Publishing Company, Incorporated, 2015.

Nikita Spirin and Jiawei Han, Survey on web spam detection:Principles and algorithms, SIGKDD Explor. Newsl. 13 (2012),no. 2, 50–64.

Claude Elwood Shannon, A mathematical theory ofcommunication, ACM SIGMOBILE Mobile Computing andCommunications Review 5 (2001), no. 1, 3–55.

Igor Santos, Igor Miñambres-Marcos, Carlos Laorden, PatxiGalán-García, Aitor Santamaría-Ibirika, and Pablo GarcíaBringas, Twitter content-based spam filtering, pp. 449–458,Springer International Publishing, Cham, 2014.


43

References III

Kurt Thomas, Chris Grier, Dawn Song, and Vern Paxson,Suspended accounts in retrospect: An analysis of twitter spam,Proceedings of the 2011 ACM SIGCOMM Conference onInternet Measurement Conference (New York, NY, USA), IMC’11, ACM, 2011, pp. 243–258.

Mahdi Washha, Aziz Qaroush, and Florence Sedes, Leveragingtime for spammers detection on twitter, Proceedings of the 8thInternational Conference on Management of Digital EcoSystems(New York, NY, USA), MEDES, ACM, 2016, pp. 109–116.


identifying twitter spam by utilizing random forests · 1 introduction i top social media platforms...

Documents