the scraping problem in ticketing - martin zetterlund from sentor
TRANSCRIPT
THE SCRAPING
PROBLEM
IN TICKETING
Martin Zetterlund,
Founder of Sentor Managed Security Services
SCRAPING DEFINED
“To automatically interact with a
website through the use of a computer
program.“
We also include manual labor in low
cost countries that are sometimes
utilized to do the work of a computer
program
HOW COMMON IS IT
• Our clients averaged 23 % of
scraped traffic.
• Those hardest hit experienced in
excess of 50 % of scraping traffic to
their sites.
Data from The Scraping Threat Report 2014
SCRAPING: AN INDUSTRY
“The companies stealing your data or scraping your tickets might have more
resources than you expect and they change their behavior in order to bypass
safety methods”
Examples of professional scraping companies that sell their services to
anyone are screenscraper.com and mozenda.com
THE LEGAL SITUATION
• Unclear and fairly complicated
situation. Ticketmaster seem to
have had some success in US and
have an interesting case against
higs tickets right now.
• O‟leary won the right to call resellers
„worthless ticket touts‟ in ads.
OUR CLIENT
• Major ticket vendor based in London
with a world-wide reputation
• Experienced issues with waiting
room engaging due to scrapers
• Negative publicity as well as site
overload
THE PROBLEM
Second hand sales
• Half empty venues
due to tout having
problem selling
tickets
• Bad PR
Site overload
• Will cause more
expensive
infrastructure
• Bad user experience
Functionality issues
• Seats/tickets
unavailable to users
due to bots
occupying them by
use of website
functionality
LESSONS LEARNED
Our client attempted to solve the problem by using manual log analysis to find
IP addresses that hit their site or seemed automated in their behavior.
Time consuming
Reactive
No automated blocking
Old data logs
HOW TO STOP SCRAPING
• Rate limiting
Against non-distributed scrapers
• Captcha
Effects user experience but can
be effective against scrapers
• Blacklisting
IP addresses, User-agents
http://www.scrapesentry.com for
more details
KEY SUCCESS FACTORS
Resources
Needs both dev and
analysis to work together
over time
Technology
Platform needs to be
able to adapt and
evolve over time
Persistence
Problem will not go
away. Staff for long
time.
SCRAPESENTRY IN BRIEF
BENEFITS OF SCRAPESENTRY
• Correlates scraper IP information over a
global network of clients
• Real-time analysis of user behaviour on
website in order to identify scrapers
• Continuously developed to stay ahead
of scrapers
• Supported 24/7 by security analysts and
security operators from the Sentor
Security Operations Centre.
