the good bad - akamai · • java checkers • captchas • rate controls • validation on...

Statistics based on attack campaigns mitigated by Akamai

Poorly-coded bots can harm site performance and increase the load on web serversPE

RFO

RMA

NCE

LOAD

FraudMobile apps

Fraudapps

FraudStatistics

Financial statements

MetasearchMobileMetasearchMobile

WHAT THEY DO:WHAT THEY DO:

GOOD BOTS & BAD BOTS

Data mashupsStatistics

Data mashupsStatisticsPortfolio analysis

Data mashupsPortfolio analysis

Data mashups

Competitive intelligence

Location trackingHotel rates & airfares

www.akamai.com/stateoftheinternet-security

1. Volume of requests2. Type of scraped content3. User agent info

ways toidentify client reputation for bots, spiders and scrapers3

Highly desired, low aggression• Helps users• Respects robots.txt• Not too many requestsSolution• Offer an API• Dedicated data feed

Undesired, highly aggressive• Poor error handling• GET and POST floods• Thousands of requests/secondSolution• Blocklists• Rate controls• Tar pits• Spider traps• Protect login pages

Highly desired, high aggression• Access to emerging markets• Baidu-bots• Poor request throttlingSolution• Rate controls with high thresshold• User-prioritization application

Low desirability, low aggression• Scrape content for reuse• Fraud & counterfeiting• Headless browsers• Scraping-as-a-serviceSolution• Java checkers• CAPTCHAs• Rate controls• Validation on sensitive pages• Validation for suspicious IPs

DESIRABILITY

AG

GRE

SSIV

ENES

S

HIGH

HIGH

LOW

LOW

Website Copier

Baidu-bot

Googlebot

LocationScraper

Product pagescraper

Metasearchscraper

BOTS, SPIDERS AND SCRAPERS:

&&&THE

GOODGOODTHE

BAD

&&& THEUGLY