From Captchas to Captchæckers:

Can we automate usability and

security evaluation of CAPTCHAs?

Shujun LI (李树钧)

Department of Computing

University of Surrey

http://www.hooklee.com

2

Starter 1: SONY CAPTCHA

- CAPTCHA @ SONY web forum (2011)

- In Google Chrome 21.0.1180.75 m:

- In Mozilla Firefox 15.0.1:

- In MSIE 9.0.8.112.16421:

- It is obviously weak, but…

3

Starter 2: an e-banking CAPTCHA

- CAPTCHA @ a Chinese bank’s e-banking login

Web page

- In all web browsers:

- It seems to be better than the previous one, but is not

really strong. However, the simplest way of breaking it is

… 5555555555555555555555555555555555555555555455555555555555555555555555555555555555555555555555555555

5555555555555555555555555555555555555551555545555555455555555555555555555555555555555555555555555555

5555555555555555555555555555555555555511555555555554555555555555555555555555555555555555555555555555

5555555555555555555555555555555555555115555555555555555555555555555555555000000005555555555555555555

5555545555555555555555555555554155555115555555555555333555555555555555500000000005555555555555555555

5555544222225555555555555555511445551155555555555555333333555555555555505555550005555555555555555555

5555542222222255555555555555551114551155555555555555333333335555555555555555500055555555555555555555

5555522255552255555555555555551111511555555555555553335555333555555555555555500055555555555555555555

5555522255552225555555555455555111511555555555555553335555333555555555555555000555555555555555555555

5555522255455222555555555545555111115555555555555553335555333555555555555555000455555555555555555555

5555522254445222555555555554555511115555555555555553333333333555555555555550005555555555555555555555

5555552225555222555555555555455511155555555555555553333333355555555555555550005555555555555555555555

5555552222552222555555555555555551155555555555555553335533355555555555555500005555555554555555555555

5555555522222222555555555555555551155555555555555553335553335555555555555500055555555445555555555555

5555555555555222555555555555555511155555555555555553335555333555555555555000055555554555555555555555

5555555555555222555555555555555511155555555555555553335555333555555555555000555500055555555555555555

5555555555555222555555555444455511155555555555555533335553333555555555544000000000055555555555555555

5555555254452225555555555555555511155555555555555553333333333555555555555440005555555555555555555555

5555555222222225555555555555555515555555555555555555555333335555555555555555555555555555555555555555

5555555552222555555555555555555555555555555555555555555555555555555555555555555555555555555555555555

5555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555

5455555555555555555555555555555555555555544455555555555555555555555555555555555555555555555555555555

5545555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555455555

5545555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555455555

5555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555545555

5555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555545555

5555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555554555

5555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555

4

Starter 3: CAPTCHA @ a Chinese site

- “Input the result of executing the above code

________ refresh the page to get other code”.

5

More starters: top 10 worst CAPTCHAs

- No. 1:

- No. 2:

- No. 3:

6

The main dishes

- Captchas (or CAPTCHAs)

- Security-Usability Dilemma

- Insecure but more usable CAPTCHAs

- Secure (?) but less usable CAPTCHAs

- CAPTCHA security and usability evaluation: what is the

current practice?

- Captchæckers (ongoing research)

- Automating usability evaluation

- Automating security evaluation

- Automating data collection

- Questions and Answers

Captchas Captchæckers

Captchas (or CAPTCHAs)

8

What are Captchas (or CAPTCHAs)?

- CAPTCHA

- Completely Automated Public Turing test to tell

Computers and Humans Apart

- It was proposed to fight against automated programs

abusing web resources (e.g. spamming).

I am human!

Then solve this!

9

CAPTCHA has many names!

- CAPTCHA: A Turing test?

- Automated Turing Test? – The human interrogator in a

Turing test is automated by a computer.

- Reversed Turing Test? – The role of something (human

interrogator) is reversed in a Turing test.

- CAPTCHA = HIP (Human Interactive Proof)?

- Historically, Blum et al. coined the term HIP to cover

many human-involved security systems including

CAPTCHA and HumanOID.

- So, CAPTCHA HIP.

- CAPTCHA = Authentication code?

- …

10

CAPTCHA: before the term was coined

- Moni Naor, Verification of a human in the

loop or identification via the Turing test, 1996

- , “Add-URL” web page, protected by

a scheme later known as CAPTCHA, 1997

- US Patent 6195698, Method for

selectively restricting access to computer

systems, filed on 13 April, 1998, issued on 27

February, 2001

- Jun Xu, Richard Lipton and Irfan Essa, Hello,

Are You Human? Georgia Institute of

Technology College of Computing Technical

Report, GIT-CC-00-28, 13 November 2000

11

CAPTCHA: after the term was coined

- 2000: Udi Manber from described the

“chat room problem” to Manuel Blum at the UC

Berkeley (who later moved to the CMU).

- 2000-2003: Blum and his collaborators coined the

term “CAPTCHA” and proposed some early

designs at www.captcha.net.

- 2002: the first report on

breaking CAPTCHAs appeared.

- 2002 onwards: a new kind of

cat-and-mouse game…

12

CAPTCHAs everywhere

- Many (most?) user registration web pages are

protected by CAPTCHAs.

- Many login pages and web forms as well.

13

CAPTCHAs everywhere

- CAPTCHA (reCAPTCHA) has been used for

digitizing books by Google.

14

CAPTCHAs everywhere

- CAPTCHA has been used as a new advertising

platform as well!

15

CAPTCHAs everywhere

- SweetCaptcha

- PlayThru

- MotionCAPTCHA

Captchas Captchæckers

Security-Usability Dilemma

17

Security-usability dilemma

- Let’s look at our good friend: (textual) passwords.

- Dinei Florêncio and Cormac Herley, A Large-Scale

Study of Web Password Habits, in Proc. WWW 2007,

ACM/W3C

18

Security-usability dilemma

- Let’s look at our good friend: (textual) passwords.

- DataGenetics, PIN analysis, 3 September 2012

xy00

xy9900xy 19xy

mmdd

xyxy

19

Security-usability dilemma

- So for passwords the dilemma is

- If a password is very strong (secure), then it

is not usable (hard to remember).

- If a password is usable (easy to remember),

then it is very weak (insecure).

- If I have to use a strong password but cannot

remember it, I will write it down!

- There is a similar one for CAPCTAHs!

- If a CAPTCHA is strong (hard for machines),

then it is hard to solve by humans.

- If a CAPTCHA is easy for humans to solve, it

is often weak (i.e., easy for machine as well).

20

Insecure but usable CAPTCHAs

- Almost all (if not all) e-banking CAPTCHAs [S. Li

et al. ACSAC 2010]

21

Strong but less usable CAPTCHAs

- Google CAPTCHA (not reCAPTCHA)

- Simplest are not very hard to solve

- Averagely OK?

- Some are very hard (if not impossible) to solve

- Google has replaced this CAPTCHA by reCAPTCHA for

user registration, but still keep it for login (only after

three continuous login errors occur).

22

CAPTCHA security mixed with usability

- Attackers are recruiting humans to do the job!

23

CAPTCHA security mixed with usability

- Attackers also know how to recruit humans without

even paying them a penny (since 2007)!

24

CAPTCHA usability evaluation

- So far, usability of CAPTCHAs is evaluated by

recruiting real human users.

- Problems?

- Time-consuming

- Scale-cost dilemma

- Hard to reproduce?

- …

25

CAPTCHA security evaluation

- Security evaluation = Attack discovery

- Is security evaluation easy?

- Yes: if a CAPTCHA is weak, it is easy.

- No: if a CAPTCHA is strong, it can be very difficult.

- It depends on the security analyst’s experience!

- Problems?

- Slight change of CAPTCHA design may require big

changes to source code of the attack.

- Even if changes are not heavy nor difficult, it’s boring

and error-prone to do them all by hand!

- Porting from one programming language to the other

can be difficult.

26

CAPTCHA security evaluation

- Remarks of a reCAPTCHA cracker [Chad W.

Houck, DEFCON’2010]

- “Unfortunately any CAPTCHA that can be read by a

human can eventually be read by a computer. The only

solution is to stay one step ahead of those wishing to

abuse these systems by consistently changing the

CAPTCHA distortions and design. While it may take

the maintainer of a CAPTCHA system a couple of

hours to implement a change, it takes a human no

time to adjust to the difference, while a person

wishing to keep their automated system working

that defeats the CAPTCHA may take weeks to adopt

the changes necessary to get it running again.”

27

Multi-CAPTCHA engines

- Some web sites (or CAPTCHA service providers)

have started deploying multi-CAPTCHA engines.

- The idea is simple:

- include a (large) number of different CAPTCHA

schemes in the engine;

- (randomly) select a scheme to generate each new

CAPTCHA;

- the CAPTCHA poll and selection rule may be

customized by the user.

- The consequence

- Security and usability evaluation complexity

28

Multi-CAPTCHA engines

- One example

29

Multi-CAPTCHA engines

- Another example: Microsoft live.com CAPTCHAs?

- An insider told me a powerful multi-CAPTCHA engine

was (is?) in place which is able to produce many

different types of CAPTCHAs.

- I didn’t observe a diversity of CAPTCHA on live.com

user registration Web page.

(reCAPTCHA, 2010, 2011-2012?)

(2011, 2012?)

(2012, also in 2010-2011?)

- The insider explained the Microsoft staff who was

managing the engine has trouble (re-)reconfiguring it!

30

Security vs. usability

- A balance between security and usability

- No usability evaluation, no security evaluation

- (Automated) usability+security evaluation

- effort a legitimate user needs to solve a CAPTCHA

- costs of human solver based attacks

- (automatic) usability control of CAPCTHAs

- (automatic) reconfiguration of CAPTCHAs

- …

31

Now come the research questions…

- Can we automate the security and usability

evaluation of CAPTCHAs?

- If full automation is not possible, can we automate

part of the process?

- What techniques do we need?

- How do we validate results of automated

evaluation?

- How can we link automated security and usability

evaluation?

- …

Captchas Captchæckers

Captchæckers: The solution?

(Ongoing/Incomplete research)

33

What is a Captchæcker?

- Captchæcker = Captcha + checker

- It is a term created by my collaborators and me in

2011.

- A Captchæcker = A fully or partly automated

programme that can evaluate one or more

performance aspects of CAPTCHAs

- We consider security and usability only in this research.

- Captchæckers a benchmarking toolbox for

CAPTCHA performance evaluation

- Captchæckers a CAPTCHA reconfiguration

toolbox as well!

34

Usability Captchæcker

- The input: one CAPTCHA

- The output: usability metric(s) of the input

- Hardness as the first impression perceived by an

average human user: subjective

- Hardness as an experience perceived by an average

human user after trying to solve it: subjective

- Hardness measured as the average response time and

error rate of the target user population: objective

- The response time may be misleading if a user gives

up earlier (so response time is related to error rate)

- The key research question: is there a predictable

pattern in the average behavior of human users?

35

Usability Captchæcker: preliminary work

- Our positive answer via a small-scale user study

[S. Li et al., SOUPS 2011 poster]

- An artificial neutral network predicted hardness of 38

CAPTCHAs perceived by 5 users with >80% accuracy.

- Only four simple geometric features are involved.

4 6 8 10 12 14 16 18 20-9

-8

-7

-6

-5

-4

-3

-2

-1

0CL and ET Values and User Rating

Compact-Length (CL)

Eu

ler-T

hic

kn

ess

(E

T)

Extremely Easy

Somewhat Easy

Somewhat Difficult

Difficult but Readable

Impossible to Read4 6 8 10 12 14 16 18 20

-5

-4.5

-4

-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0CL and ET Values and User Rating

Compactness-Length (CL)

Eu

ler-

Th

ick

nes

s (E

T)

Extremely Easy

Somewhat Easy

Somewhat Difficult

Difficult but Readable

36

Usability Captchæcker: future work

- Add pre-processing steps (e.g. denoising)

- Increase the number of CAPTCHAs trained/tested

- Increase the number of human users involved and

the diversity of users (cultural background, age,

computer knowledge, etc.)

- Test more geometric and non-geometric features

- Try to predict all three editions of hardness

- Try more machine learning algorithms

- Hierarchical classification, ensemble methods, …

- …

37

Security Captchæcker

- The input: a number of CAPTCHAs generated by

the same scheme

- The output: security metric(s) of the input

- For each attack, the metric reports how strong the

CAPCHA scheme is against this specific attack

(percentage of fully recognized CAPTCHAs)

- The key questions: how to implement different

attacks more effectively and how to discover new

unknown attacks?

38

Security Captchæcker

- Some previous work showed attacks on many different

CAPTCHAs are based on the same set of techniques [S. Li

et al., ACSAC 2010]

- Another one: Bursztein et al., Text-based CAPTCHA

Strengths and Weaknesses, in Proc. ACM CCS 2011

Morphological

Operations

Line Detection

Image

Inpainting

Genuine

CAPTCHA images k-means Layer

Segmentation

Character

Segmentation

Character

Recognition

CAPTCHA Image

Synthesis

Image

Inpainting

Forged

CAPTCHA image

39

Security Captchæcker

- The full attack space is prohibitively huge.

- Full automation is (computationally) impossible.

- Partial automation should be possible, where the

human expert needs to define a space of attacks.

- The space of attacks may be developed using the

new (ISO/IEC) standardized dataflow

programming framework RVC (Reconfigurable

Video Coding, to be extended to RMC =

Reconfigurable Multimedia Coding).

40

What is RVC?

- A new ISO/IEC standard made by MEPG for

developing complicated multimedia codecs

Actions

State

Variables

Implementation-

Independent Design

C

Java

LLVM

VHDL

Verilog

C++

Target

Implementations

41

Automating data collection

- A database of CAPTCHAs

- How to select CAPTCHAs?

- How to extract CAPTCHAs from web pages?

- Automating the selection and extraction processes (A

Web crawler for CAPTCHAs) is very useful.

- A database of CAPTCHA usability data

- Collecting data from real human users is costly and time

consuming.

- Crowdsourcing can help?

- Automating the data collection process from

crowdsourcing Web sites will be very useful.

42

Captchæckers for whom?

- Researchers

- Who want to have a deeper understanding of different elements in

the system.

- End users

- Who want to know more about CAPTCHAs they’re using.

- Webmasters

- Who want to select a right CAPTCHA scheme for their Web sites.

- CAPTCHA service providers

- Who want to serve their customers better and improve their

products.

- CAPTCHA solving service providers

- Who want a more accurate estimate of the costs for better pricing.

- …

43

Captchæckers for end users

- Only evaluation

Security

Captchæcker

Usability

Captchæcker

Overall

metric(s)

Crowd-

sourcing

CAPTCHA

Usability

Database

CAPTCHA-

Breaking

Tool Library

CAPTCHA(s)

CAPTCHA

Database

CAPTCHA

Web Crawler

44

Captchæckers for Webmasters

- Evaluation + reconfiguration [S. Li et al.,

SafeConfig 2011]

PRNGCAPTCHA

Engine

Security

Captchæcker

Usability

Captchæcker

Reconfigurator

Crowd-

sourcing

CAPTCHA

Usability

Database

CAPTCHA-

Breaking

Tool Library

CAPTCHA

Database

Side Information

CAPTCHAs

Own Web

site(s)

CAPTCHA

Web Crawler

45

Can we go beyond CAPTCHAs?

- CAPTCHA is just one kind of computer security

system involving human users.

- Can we automate security and usability evaluation

of other human-involved computer security

systems? And if so how?

- Other HIPs? (HumanOID?)

- User authentication systems (e.g. graphical passwords)

- HCI of security software (firewall, anti-virus software,

etc.)

- Security warning systems (unprotected HTTP traffic,

saving passwords in cookie, suspicious web sites, etc.)

- …

Captchas Captchæckers

Thanks for your attention!

Questions + Answers

Collaborations?