Social Engineering at

ScaleSJ, CanSecWest 2018

@bodaceacat

“The views and opinions expressed in this presentation are mine and do not necessarily reflect the

official policy or position of AppNexus. Assumptions made and conclusions drawn in this presentation

are not reflective of the position of AppNexus.”

1

OUTLINE

❑ Social engineering at scale

❑ Responses

❑ Useful earlier work

❑ This changes internet?

❑ Now what do we do?2

SOCIAL ENGINEERING

AT SCALEFacebook group total_shares interactions number

of postswordcount

Facebook.com/Blacktivists 103,767,792 6,182,835 500+ 15349

Facebook.com/Txrebels 102,950,151 3,453,143 500+

Facebook.Com/MuslimAmerica 71,355,895 2,128,875 500+ 6892

Facebook.Com/Patriototus 51,139,860 4,438,745 500+ 53782

Facebook.Com/Secured.Borders 5,600,136 1,592,771 500+ 53850

Facebook.Com/Lgtbun 5,187,494 1,262,386 500+ 15020

3

ECOSYSTEM: INTERNET

4

ENDPOINTS: WETWARE

5

VULNERABILITIES

6

COMMONLY USED

• Imperfect recall

•Unconscious bias

•Confirmation bias

•Mental immune systems

•Familiarity backfire effect

•Memory traces

•Emotions = stronger traces

7

PAYLOADS

Misinformation Disinformation Abuse “Normal”

Accidental / sloppy

untruths

Deliberate /

focussed

untruths

Focussed opinion

/ facts Hate

speechBullying

Doxxing

Everything

else

8

CHANNELS

9

ACTORS AND AIMS

10

Let’s make that concrete...

11

RESPONSES

12

CRISISMAPPERS

13

ADTECH

14

MEDIA / JOURNALISTS

15

POLITICS I

16

POLITICS II

17

POLITICS III

18

SOCIAL NETWORKS

19

HACKERS

Build stuff! Add extra machine learning!

20

USEFUL EARLIER WORK

(Those Darned Telegraph

Operators…)

21

BIG DATA’S FOURTH V

22

WIKIPEDIA

23

HACKERS, OSINT

24

AI MODEL POISONING

● Bad inputs

○ Biased classifications

○ Missing demographics

● Bad models

○ Unclean inputs, assumptions etc

○ Lazy interpretations (eg. clicks == interest)

○ Trained once in a changing world

● Willful abuse

○ gaming with ‘wrong’ data (propaganda etc)

○ gaming with adversarial data

25

AI MODELLING HUMANS

26

THIS CHANGES INTERNET?

27

PEOPLE SHIFTS

28

BOTS AREN’T GOING

AWAY

29

FUNDING SHIFTS

30

NEW HACK LAYER/

INDUSTRY FORMING

31

TRUST AS A COMMODITY

32

NOW WHAT DO WE DO?

33

COEXIST WITH BOTS

34

USE CRISIS MANAGEMENT

• Prevention - e.g. change structures

• Protection - e.g. set boundaries

• Mitigation - e.g. set alerts and responses

• Response - e.g. remove bots

• Recovery - e.g. rebuild community, trust

35

STOP JUST REACTING

36

DESIGN FOR BELIEF

HACKING

Set policies (cf spam and hate speech

policies)

Remove incentives (attention, money, effects)

Design for communities

Build protections into systems

37

RESPOND AT

APPROPRIATE SCALE

• Nationstate

• Platform

• Organisation

• Community

• Individual

38

SOME PRACTICAL STUFF

39

DO DO THIS AT HOME

● https://twitter.com/probabot_ - BotOrNot

● https://data.world/d1gi/ - Troll tweets, facebook, youtube

● http://library.sewanee.edu/fakenews - ‘fakenews’ sites list

40

BOT/TROLL HUNTING

41

RED BOT/ TROLL

• Evasion:

• stagger creation dates

• buy or take over old dead accounts

• use dictionary or NLP to vary output text

• use api to push tweets out in e.g. Russian nighttime

• test variants on e.g. botornot

• Manipulation:

• Find high-volume, high-sentiment topics

• Microtarget (e.g. adint), and wipe artefacts immediately

42

DISINFORMATION

HUNTING

• “Content” signals (cf “context” signals)

• Repeated information (cut n pastes)

• Bots repeating or pointing at it!

• No (or very little) original text

• Large numbers of ad slots

• Clickbait words

43

RED DISINFORMATION

• Vary text (enough to defeat shingling)

• Mix nudges with truths

• Context matters. Format matters

• Talk about something else goddammit

• Microtarget: use network analysis and probes

• Keep using current vulnerabilities

44

Social Engineering at

ScaleSJ, CanSecWest 2018

@bodaceacat

45