freezing the web: a study of redos vulnerabilities in ......using node.js redos analysis of websites...

Freezing the Web: A Study of ReDoSVulnerabilities in JavaScript-based Web Servers

Cristian-Alexandru Staicu Michael Pradel

TU Darmstadt

www.software-lab.org

15th August 2018

1/18

Regular Expression Denial of Service (ReDoS)

input: ”Lorem ipsum”

input.match(regexp);

processing time: O(1)

input: ”aoAs,”x 1000

processing tim

e: O(nx ), x

> 1

1/18

Regular Expression Denial of Service (ReDoS)

input: ”Lorem ipsum”

input.match(regexp);

processing time: O(1)

input: ”aoAs,”x 1000

processing tim

e: O(nx ), x

> 1

1/18

Regular Expression Denial of Service (ReDoS)

input: ”Lorem ipsum”

input.match(regexp);

processing time: O(1)

input: ”aoAs,”x 1000

processing tim

e: O(nx ), x

> 1

1/18

Regular Expression Denial of Service (ReDoS)

input: ”Lorem ipsum”

input.match(regexp);

processing time: O(1)

input: ”aoAs,”x 1000

processing tim

e: O(nx ), x

> 1

1/18

Regular Expression Denial of Service (ReDoS)

input: ”Lorem ipsum”

input.match(regexp);

processing time: O(1)

input: ”aoAs,”x 1000

processing tim

e: O(nx ), x

> 1

1/18

Regular Expression Denial of Service (ReDoS)

input: ”Lorem ipsum”

input.match(regexp);

processing time: O(1)

input: ”aoAs,”x 1000

processing tim

e: O(nx ), x

> 1

1/18

Regular Expression Denial of Service (ReDoS)

input: ”Lorem ipsum”

input.match(regexp);

processing time: O(1)

input: ”aoAs,”x 1000

processing tim

e: O(nx ), x

> 1

2/18

This Talk

ReDoS affects librarieswe identify 25 vulnerabilities in popular npm modules

ReDoS affects websiteshundreds of live websites are vulnerable

Novel methodologylibrary vulnerability → website vulnerability

2/18

This Talk

ReDoS affects librarieswe identify 25 vulnerabilities in popular npm modules

ReDoS affects websiteshundreds of live websites are vulnerable

Novel methodologylibrary vulnerability → website vulnerability

2/18

This Talk

ReDoS affects librarieswe identify 25 vulnerabilities in popular npm modules

ReDoS affects websiteshundreds of live websites are vulnerable

Novel methodologylibrary vulnerability → website vulnerability

3/18

Backtracking-based Matching

var regEx = /^a*a*b$/;

start 3 4 5 6

accept

7

891011

ε

ε ε

a

εε

bε ε

ε

ε ε

a

ε

3/18

Backtracking-based Matching

var regEx = /^a*a*b$/;

start 3 4 5 6

accept

7

891011

ε

ε ε

a

εε

bε ε

ε

ε ε

a

ε

3/18

Backtracking-based Matching

var regEx = /^a*a*b$/;

start 3 4 5 6

accept

7

891011

ε

ε ε

a

εε

bε ε

ε

ε ε

a

ε

input: ”ab”

3/18

Backtracking-based Matching

var regEx = /^a*a*b$/;

start 3 4 5 6

accept

7

891011

ε

ε ε

a

εε

bε ε

ε

ε ε

a

ε

input: ”aab”

3/18

Backtracking-based Matching

var regEx = /^a*a*b$/;

start 3 4 5 6

accept

7

891011

ε

ε ε

a

εε

bε ε

ε

ε ε

a

ε

input: ”aab”

3/18

Backtracking-based Matching

var regEx = /^a*a*b$/;

start 3 4 5 6

accept

7

891011

ε

ε ε

a

εε

bε ε

ε

ε ε

a

ε

input: ”aaaaaaaaaaaaaaaaaaaa”

3/18

Backtracking-based Matching

var regEx = /^a*a*b$/;

start 3 4 5 6

accept

7

891011

ε

ε ε

a

εε

bε ε

ε

ε ε

a

ε

input: ”aaaaaaaaaaaaaaaaaaaa”

4/18

Overview

Npm modules (Phase 1)ReDoS analysisof libraries

Exploits creation

Module levelvulnerabilities

Usage scenarios (Phase 2)

List of websitesusing Node.js

ReDoS analysisof websites

Payloads usingHTTP requests

List of vulner-able websites

(Phase 3)

Local machines

Live websites

4/18

Overview

Npm modules (Phase 1)ReDoS analysisof libraries

Exploits creation

Module levelvulnerabilities

Usage scenarios (Phase 2)

List of websitesusing Node.js

ReDoS analysisof websites

Payloads usingHTTP requests

List of vulner-able websites

(Phase 3)

Local machines

Live websites

4/18

Overview

Npm modules (Phase 1)ReDoS analysisof libraries

Exploits creation

Module levelvulnerabilities

Usage scenarios (Phase 2)

List of websitesusing Node.js

ReDoS analysisof websites

Payloads usingHTTP requests

List of vulner-able websites

(Phase 3)

Local machines

Live websites

4/18

Overview

Npm modules (Phase 1)ReDoS analysisof libraries

Exploits creation

Module levelvulnerabilities

Usage scenarios (Phase 2)

List of websitesusing Node.js

ReDoS analysisof websites

Payloads usingHTTP requests

List of vulner-able websites

(Phase 3)

Local machines

Live websites

5/18

Setup

measure in single instancesetup

analyze 2,800 websites fromTop 1 million

manually analyze popularpackages

fifth most-dependent uponnpm package

6/18

Node.js Particularities

Event Loop (JS code)

e1

e2

e3

...

file system

network

process

...

Events Workers

blocking I/O

callback

Regex.match()

6/18

Node.js Particularities

Event Loop (JS code)

e1

e2

e3

...

file system

network

process

...

Events Workers

blocking I/O

callback

Regex.match()

7/18

Node.js Particularities (2)

Node.jsapplication

stringsutility

templatesengine

DBaccess

headersparser

vulnerablemodule

...

...

...

8/18

Ethical Considerations

Few payloads80 requests in total

Iterative probingmost websites use redundancy

Safety mechanismstop after timeout or error

Vulnerabilities disclosurethe majority of them have been fixed

8/18

Ethical Considerations

Few payloads80 requests in total

Iterative probingmost websites use redundancy

Safety mechanismstop after timeout or error

Vulnerabilities disclosurethe majority of them have been fixed

8/18

Ethical Considerations

Few payloads80 requests in total

Iterative probingmost websites use redundancy

Safety mechanismstop after timeout or error

Vulnerabilities disclosurethe majority of them have been fixed

8/18

Ethical Considerations

Few payloads80 requests in total

Iterative probingmost websites use redundancy

Safety mechanismstop after timeout or error

Vulnerabilities disclosurethe majority of them have been fixed

8/18

Ethical Considerations

Few payloads80 requests in total

Iterative probingmost websites use redundancy

Safety mechanismstop after timeout or error

Vulnerabilities disclosurethe majority of them have been fixed

9/18

Phase 1: Npm Analysis

Criterion for vulnerable libraries

We consider a module to be vulnerable iff we find an input that

is at most 80,000 characters long,

whose matching time takes more than 5 seconds.

Manual analysis of regular expressions and information flow

Manually written exploits

9/18

Phase 1: Npm Analysis

Criterion for vulnerable libraries

We consider a module to be vulnerable iff we find an input that

is at most 80,000 characters long,

whose matching time takes more than 5 seconds.

Manual analysis of regular expressions and information flow

Manually written exploits

9/18

Phase 1: Npm Analysis

Criterion for vulnerable libraries

We consider a module to be vulnerable iff we find an input that

is at most 80,000 characters long,

whose matching time takes more than 5 seconds.

Manual analysis of regular expressions and information flow

Manually written exploits

10/18

Phase 1: Vulnerable Regular Expressions

25 ReDoS vulnerabilities

13 advisories

One bug bounty

Example 1: content

/^([^\/]+\/[^\s;]+) (?:(?:\s*;\s*boundary =(?:"([^"

]+)"|([^;"]+)))|(?:\s*;\s*[^=]+=(?:(?:"(?:[^"

]+)")|(?:[^;"]+))))*$/i

Example 2: ua-parser-js

/ip[honead ]+(.* os\s([\w]+)*\ slike\smac |;\ sopera)/

10/18

Phase 1: Vulnerable Regular Expressions

25 ReDoS vulnerabilities

13 advisories

One bug bounty

Example 1: content

/^([^\/]+\/[^\s;]+) (?:(?:\s*;\s*boundary =(?:"([^"

]+)"|([^;"]+)))|(?:\s*;\s*[^=]+=(?:(?:"(?:[^"

]+)")|(?:[^;"]+))))*$/i

Example 2: ua-parser-js

/ip[honead ]+(.* os\s([\w]+)*\ slike\smac |;\ sopera)/

10/18

Phase 1: Vulnerable Regular Expressions

25 ReDoS vulnerabilities

13 advisories

One bug bounty

Example 1: content

/^([^\/]+\/[^\s;]+) (?:(?:\s*;\s*boundary =(?:"([^"

]+)"|([^;"]+)))|(?:\s*;\s*[^=]+=(?:(?:"(?:[^"

]+)")|(?:[^;"]+))))*$/i

Example 2: ua-parser-js

/ip[honead ]+(.* os\s([\w]+)*\ slike\smac |;\ sopera)/

10/18

Phase 1: Vulnerable Regular Expressions

25 ReDoS vulnerabilities

13 advisories

One bug bounty

Example 1: content

/^([^\/]+\/[^\s;]+) (?:(?:\s*;\s*boundary =(?:"([^"

]+)"|([^;"]+)))|(?:\s*;\s*[^=]+=(?:(?:"(?:[^"

]+)")|(?:[^;"]+))))*$/i

Example 2: ua-parser-js

/ip[honead ]+(.* os\s([\w]+)*\ slike\smac |;\ sopera)/

10/18

Phase 1: Vulnerable Regular Expressions

25 ReDoS vulnerabilities

13 advisories

One bug bounty

Example 1: content

/^([^\/]+\/[^\s;]+) (?:(?:\s*;\s*boundary =(?:"([^"

]+)"|([^;"]+)))|(?:\s*;\s*[^=]+=(?:(?:"(?:[^"

]+)")|(?:[^;"]+))))*$/i

Example 2: ua-parser-js

/ip[honead ]+(.* os\s([\w]+)*\ slike\smac |;\ sopera)/

11/18

Phase 2: HTTP-level Payload Creation

Local Node.js installation

For each payload, create a usage scenario

var MobileDetect = require("mobile -detect");

var headers = req.headers["user -agent"];

var md = new MobileDetect(headers);

md.phone();

For each scenario, create HTTP level payloads

In total 8 payloads corresponding to 8 popular modules

11/18

Phase 2: HTTP-level Payload Creation

Local Node.js installation

For each payload, create a usage scenario

var MobileDetect = require("mobile -detect");

var headers = req.headers["user -agent"];

var md = new MobileDetect(headers);

md.phone();

For each scenario, create HTTP level payloads

In total 8 payloads corresponding to 8 popular modules

11/18

Phase 2: HTTP-level Payload Creation

Local Node.js installation

For each payload, create a usage scenario

var MobileDetect = require("mobile -detect");

var headers = req.headers["user -agent"];

var md = new MobileDetect(headers);

md.phone();

For each scenario, create HTTP level payloads

In total 8 payloads corresponding to 8 popular modules

11/18

Phase 2: HTTP-level Payload Creation

Local Node.js installation

For each payload, create a usage scenario

var MobileDetect = require("mobile -detect");

var headers = req.headers["user -agent"];

var md = new MobileDetect(headers);

md.phone();

For each scenario, create HTTP level payloads

In total 8 payloads corresponding to 8 popular modules

11/18

Phase 2: HTTP-level Payload Creation

Local Node.js installation

For each payload, create a usage scenario

var MobileDetect = require("mobile -detect");

var headers = req.headers["user -agent"];

var md = new MobileDetect(headers);

md.phone();

For each scenario, create HTTP level payloads

In total 8 payloads corresponding to 8 popular modules

12/18

Phase 2: Input Dependency

0

500

1000

1500

2000

0 10000

20000

30000

40000

50000

60000

70000

Matc

hin

g t

ime

(ms)

Input size (number of characters)

ua-parser-js

12/18

Phase 2: Input Dependency

0

500

1000

1500

2000

0 10000

20000

30000

40000

50000

60000

70000

Matc

hin

g t

ime

(ms)

Input size (number of characters)

ua-parser-js

useragent

12/18

Phase 2: Input Dependency

0

500

1000

1500

2000

0 10000

20000

30000

40000

50000

60000

70000

Matc

hin

g t

ime

(ms)

Input size (number of characters)

charset

fresh

forwarded

content

mobile-detect

platform

ua-parser-js

useragent

13/18

Phase 3: Websites Analysis

3x

3x

5x

5x

100ms

P1

3x

3x

5x

5x

200ms

P2

3x

3x

5x

5x

500ms

P3

3x

3x

5x

5x

1s

P4

3x

3x

5x

5x

2s

P5

13/18

Phase 3: Websites Analysis

3x

3x

5x

5x

100ms

P1

3x

3x

5x

5x

200ms

P2

3x

3x

5x

5x

500ms

P3

3x

3x

5x

5x

1s

P4

3x

3x

5x

5x

2s

P5

13/18

Phase 3: Websites Analysis

3x

3x

5x

5x

100ms

P1

3x

3x

5x

5x

200ms

P2

3x

3x

5x

5x

500ms

P3

3x

3x

5x

5x

1s

P4

3x

3x

5x

5x

2s

P5

13/18

Phase 3: Websites Analysis

3x

3x

5x

5x

100ms

P1

3x

3x

5x

5x

200ms

P2

3x

3x

5x

5x

500ms

P3

3x

3x

5x

5x

1s

P4

3x

3x

5x

5x

2s

P5

Criterion for vulnerable websites

We consider a website to be vulnerable iff:

statistically significant difference between the response timesto random and crafted inputs,

this difference increases when the input size increases.

14/18

Phase 3: Response Time of a Non-Vulnerable Website

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

P1

P2

P3

P4

P5

Res

ponse

tim

e (m

s)

Payload number (increasing in size)

Random Crafted

15/18

Phase 3: Response Time of a Vulnerable Website

0

500

1000

1500

2000

2500

P1

P2

P3

P4

P5

Res

ponse

tim

e (m

s)

Payload number (increasing in size)

Random Crafted

16/18

Phase 3: Number of Vulnerable Websites

Exploit Number of sites affected

fresh 241forwarded 99

ua-parser-js 41useragent 16

mobile-detect 9platform 8charset 3content 0

In total: 339 (11%) websites are vulnerable

16/18

Defenses

0

500

1000

1500

2000

2500

3000

1 10 100

1000

10000

100000

Num

ber

of w

ebsi

tes

Accepted header size

charset

16/18

Defenses

0

500

1000

1500

2000

2500

3000

1 10 100

1000

10000

100000

Num

ber

of w

ebsi

tes

Accepted header size

fresh

forwarded

ua-parser-js

useragent

mobile-detect

platform

charset

content

17/18

Defenses (2)

Linear time matching algorithms / hybridRust programming language

Timeout on matching regular expressions[Davis et al., USENIX Security, 2018], .NET framework

Tooling support for identifying ReDoSJava programming language [Wustholz et al., TACAS, 2017]

17/18

Defenses (2)

Linear time matching algorithms / hybridRust programming language

Timeout on matching regular expressions[Davis et al., USENIX Security, 2018], .NET framework

Tooling support for identifying ReDoSJava programming language [Wustholz et al., TACAS, 2017]

17/18

Defenses (2)

Linear time matching algorithms / hybridRust programming language

Timeout on matching regular expressions[Davis et al., USENIX Security, 2018], .NET framework

Tooling support for identifying ReDoSJava programming language [Wustholz et al., TACAS, 2017]

18/18

Conclusions

ReDoS is a widespread problem in npm modules,

Npm modules vulnerabilities can be exploited in live websites

11% of websites using Express are vulnerable to ReDoS

ReDoS vulnerabilities can be used to fingerprint web servers

More tools are needed to mitigate the ReDoS risk

18/18

Conclusions

ReDoS is a widespread problem in npm modules,

Npm modules vulnerabilities can be exploited in live websites

11% of websites using Express are vulnerable to ReDoS

ReDoS vulnerabilities can be used to fingerprint web servers

More tools are needed to mitigate the ReDoS risk

18/18

Conclusions

ReDoS is a widespread problem in npm modules,

Npm modules vulnerabilities can be exploited in live websites

11% of websites using Express are vulnerable to ReDoS

ReDoS vulnerabilities can be used to fingerprint web servers

More tools are needed to mitigate the ReDoS risk

18/18

Conclusions

ReDoS is a widespread problem in npm modules,

Npm modules vulnerabilities can be exploited in live websites

11% of websites using Express are vulnerable to ReDoS

ReDoS vulnerabilities can be used to fingerprint web servers

More tools are needed to mitigate the ReDoS risk

18/18

Conclusions

ReDoS is a widespread problem in npm modules,

Npm modules vulnerabilities can be exploited in live websites

11% of websites using Express are vulnerable to ReDoS

ReDoS vulnerabilities can be used to fingerprint web servers

More tools are needed to mitigate the ReDoS risk

18/18

Conclusions

ReDoS is a widespread problem in npm modules,

Npm modules vulnerabilities can be exploited in live websites

11% of websites using Express are vulnerable to ReDoS

ReDoS vulnerabilities can be used to fingerprint web servers

More tools are needed to mitigate the ReDoS risk

18/18

Is Response Time a Good Estimator?

50

100

150

200

250

300

350

400

450

500

0 50 100

150 200

250 300

350 400

450

Res

ponse

tim

e (m

s)

Matching time (ms)

18/18

Popularity of Vulnerable Websites

0

50

100

150

200

250

300

350

0 100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

Num

ber

of vuln

erable

web

site

s

Popularity ranking

18/18

Dimensioning Payloads

Module P1 P2 P3 P4 P5100ms 200ms 500ms 1s 2s

fresh 12,000 17,000 27,000 37,500 53,500forwarded 12,000 17,000 26,500 38,000 53,500useragent 500 650 925 1,150 1,450

ua-parser-js 38 39 40 41 42mobile-detect 10,500 15,500 25,000 36,500 50,500

platform 7,500 11,000 17,500 25,000 34,500charset 10,500 15,500 24,000 34,000 48,000content 8,000 11,000 18,000 25,500 35,500