bots to the future - frost & sullivan · 2016-03-22 · distracting, and unrewarding. many...

Bots to the Future

Bot Management Solutions Necessary to Optimize Processes, Stop Hackers

Stratecast Perspectives and Insight for Executives (SPIE) Volume 15, Number 42

Stratecast Analysis by Chris Rodriguez

Contents

Introduction ..................................................................................................................................................................... 3

Organizations Underestimate or Misunderstand the Bot Challenge .................................................................. 4

Bots: The Good, the Bad, and the Ugly ............................................................................................................................... 4

Bot Risks Vary by Industry .................................................................................................................................................... 4

Bots Cross Multiple IT Disciplines ..................................................................................................................................... 5

Current Bot Detection Solutions Fall Short ............................................................................................................. 5

Basic Control Systems are Ineffective ............................................................................................................................... 5

Bots Break Traditional Security Models ............................................................................................................................. 6

Bot Lists and Trackers are Not Security Focused ............................................................................................................ 6

Multipurpose Bots Can Defeat Systems that are Focused on a Single Problem.................................................... 6

Bot Management as an Emerging Market .................................................................................................................. 7

Defining Bot Management ................................................................................................................................................... 7

Understanding Bots as a Tool of Human Actors ............................................................................................................. 8

Security Solution Architectures that Support Bot Management ................................................................................ 8

New and Emerging Bot Management Solutions ....................................................................................................... 9

The Importance of Cloud Intelligence for Bot Management ....................................................................................... 9

Stratecast: The Last Word ............................................................................................................................................. 10

frost.com

3All rights reserved © 2016 Frost & Sullivan

Bots to the Future: Bot Management Solutions Necessary to Optimize Processes, Stop Hackers

INtroductIoN1

As of October 21, 2015, the future has arrived. As Marty McFly discovered, the world is drastically different in many ways, but disarmingly familiar in others. Flying cars and hover boards may not yet exist, but many other futuristic toys are now realities. Virtual reality systems, augmented reality devices, and virtual content are now ready for public consumption. Wearable cameras, 360-degree cameras, and camera-equipped drones are delivering more immersive experiences. Tesla is producing high-performance, fully electric-powered automobiles that outperform most of their gas-powered counterparts, and Google is developing self-driving cars. Even autonomous droids are now available for consumer purchase. Futuristic technologies are here, and it is now only a matter of time before these products are ubiquitous tools rather than elite toys.

Similarly, in the technology world, emerging technologies such as virtualization and cloud computing are quickly gaining traction in all types of networks. These technologies have multiple benefits, such as making networks more scalable and reliable, reducing costs, and increasing flexibility. However, the development of network security tools has remained very linear in at least one important area—bots. Bots are computer programs that are used to perform specific actions in automated fashion. For example, Google uses bots to categorize the Web. Enterprises use bots to interact with partners. However, this programmability and autonomy makes the bot a popular tool for malicious actors, acting as a force multiplier that enables spam distribution, malware propagation, massive distributed denial-of-service (DDoS) attacks, and numerous other hacking and fraudulent activities.

Yet, there have been no comprehensive solutions available that are tailor-made to address the unique challenges presented by bots. Bot detection capabilities are available, but have traditionally been offered as an integrated function of other security tools. In 2014, Stratecast highlighted the importance of these bot detection features as a key indicator of security efficacy in various network security solutions, such as firewalls, intrusion prevention systems (IPS), DDoS mitigation solutions, and Web application firewalls (WAF).2

The challenge of bot activity in business networks has grown in 2015, leading to the imminent emergence of dedicated bot management solutions. This SPIE will review the challenges presented by bots, emerging solutions for bot management, and the potential market for bot management.

4 All rights reserved © 2016 Frost & Sullivan


orgaNIzatIoNS uNdErEStImatE or mISuNdErStaNd thE Bot challENgE

By some estimates, bot traffic can account for up to 70% of traffic on any given website.3 This high level of bot activity is not necessarily indicative of a security problem, but does present a number of serious implications for security and network performance.

Bots: the Good, the Bad, and the UGly

To many, the term “bot” is instantly associated with malware. While it is true that threat actors utilize bots to carry out a number of malicious actions, including attacking, spamming, and scanning, it is important to remember that bots are simply tools. Many bots exist to perform valuable and highly desired functions such as Web indexing, search engine optimization (SEO), and interaction with partner systems. Bots are responsible for delivering new content to readers through Rich Site Summary (RSS) feeds, social media, and search engines. But, unwanted “bad” bots also represent a high level of all bot activity today. And some bots, such as content aggregators, are good for some companies but bad for others.

The inherent blank slate offered by autonomous bots severely complicates the “identification, friend-or-foe” (IFF) process. Businesses are interested in blocking malware, spam, and DDoS bots. But an accidental blocking of a good bot could have consequences ranging from a minor inconvenience to a catastrophic business outage. For example, blocking a Google Web crawler would remove the site from Google search results, effectively disconnecting the business from the Web.

The challenge increases exponentially when considering the many types of bots that fall in a gray area, either legally or ethically, such as click bots, scanners, comment spammers, spam distributors, and data and content aggregation bots. These functions are not illegal, but are also not entirely desirable or useful.

A common example is a travel agency, for which flight and pricing information is valuable information. Businesses spend significant resources and time to collect and assemble this data. Unfortunately, competitors can employ Web scrapers to collect this flight information and pricing data for use on their own sites. This practice is not explicitly illegal, but eliminates the competitive advantage that the victimized businesses worked to achieve. Even in cases where these activities do cross legal lines, the process for legal recourse can be expensive, tedious, distracting, and unrewarding. Many fly-by-night websites successfully leverage these legally gray business practices (enabled by bots), at the expense of more established and reputable companies.

Bot Risks VaRy By indUstRy

Sensitivity to these types of bot actions will vary drastically across vertical industries, or even from one organization to another. The financial services industry features large businesses with valuable data that are frequently attacked. These organizations are interested in eliminating unwanted bot activity. On the other hand, retail organizations must continue to enable sales at all costs. Retail and eCommerce sites are most interested in sales; any security solution that blocks potential customers, instead of bots, will be switched off.

Additionally, bots are frequently used for Web scraping, competitive intelligence collection, ad-fraud, reconnaissance, and sabotage in vertical industries such as travel, hospitality, events, and retail. These types of bot activities often fall into a moral gray area in which legal action is not a feasible option for prevention or recourse.

In some cases, the end users in a particular business or industry can exacerbate the bot problem. In online gaming, many end users identify with their peers, and see different gaming platforms, games, or groups of gamers as rivals. Many of these end users are technically savvy enough to attack competing online-gaming companies and platforms. Online gaming is now a top target for DDoS attacks, which often (but not always) leverage large


Bots to the Future: Bot Management Solutions Necessary to Optimize Processes, Stop HackersBots to the Future: Bot Management Solutions Necessary to Optimize Processes, Stop Hackers

networks of bots to consume all available bandwidth and resources in order to block access for legitimate end users. In some cases, online-gaming companies have been targeted by their own customers after an upsetting announcement, a change to gameplay mechanics or rules, or even after a frustrating loss.

The wide range of risk postures exhibited by different companies presents the biggest challenge associated with bot management and security in general: there is rarely a one-size-fits-all solution with network security.

Bots CRoss MUltiple it disCiplines

While bots are typically discussed in terms of network security, many lines of business are now interested in bot management. For example, bot traffic can provide a skewed understanding of network performance and website traffic. A company that sells event tickets might not realize that it is selling tickets to scalpers that buy tickets in bulk. The scalpers mark up prices, resulting in an artificial supply limitation and price inflation. This practice dampens the end-user experience, possibly resulting in misdirected customer frustration and mistrust toward the ticket company.

In another example, online advertisers pay per page view, and a larger number of page views should translate into increased sales. Naturally, these companies aim for human audiences rather than machine audiences. In these cases, ad-fraud bots catch the interest of marketing and business professionals, rather than security or networking teams.

As a result, bots are not always considered a network security issue. While it is useful to gain buy-in from multiple departments and decision makers, underestimating bot risk or framing bot challenges in the wrong context can result in incomplete bot management strategies.

The wide range of risk postures exhibited by different companies presents the biggest challenge associated with bot management and security in general: there is rarely a one-size-fits-all solution.

currENt Bot dEtEctIoN SolutIoNS Fall Short

Many businesses are well aware of the challenges caused by bots. Unfortunately, few options are available to manage bot traffic effectively.

BasiC ContRol systeMs aRe ineffeCtiVe

Basic systems are available to manage bots, but provide only limited controls. The most common control is the robots.txt file. Website owners can create this file with instructions for bots to define acceptable and unacceptable actions, such as which directories and files may be accessed, and which to avoid. However, the instructions in this file are completely optional, and bot developers choose whether to program bots to follow these instructions or not. The robots.txt standard offers no enforcement mechanism, and bots can be programmed to ignore these instructions.



Bots BReak tRaditional seCURity Models

Bot detection cannot be a simple binary identification process. Bot detection should also identify the type of bot, and intentions of a bot, in order to provide value. Of course, this breaks the decision model of well-known security tools such as IPS, NGFW, WAF, email security, and Web URL filtering. These tools all have binary decisions to make, including:

• Is this file malicious?

• Is this email spam malicious?

• Does this website violate policies set by human resources or information security teams?

• Is communication allowed on this protocol from this IP address?

True bot management requires granular categorization of bot types, purposes, actions, and intentions in order to support a broad range of risk postures and business objectives across many different vertical industries and numerous bot types.

Bot lists and tRaCkeRs aRe not seCURity foCUsed

Google Analytics recently announced a new service to help users better understand the amount of Web traffic generated by bots with the introduction of its bot and spider filtering service.4 However, the purpose of this service is not to eliminate bad bot traffic, but is instead offered as a method to filter network statistics in a way that provides more meaningful analytics and intelligence.

Additionally, bot lists are available for filtering use, such as the Audit Bureau of Circulations (ABC) International Spiders & Bots List.5 This list is designed to “prevent non-human traffic being counted in Web analytics.” This mission statement inherently stops at a binary “bot-or-not” decision point, which provides insufficient context for use in enacting any worthwhile controls.

Moreover, threat actors typically invest significant effort into defeating bot detection systems. Malicious bots are often designed to impersonate search engine clients and Web browsers to defeat bot detection systems. Malicious bots that impersonate good bots are on the rise, with Imperva reporting that traffic volumes from impersonator bots grew to 24.5% in 2015, up from 22% in 2014.6

MUltipURpose Bots Can defeat systeMs that aRe foCUsed on a sinGle pRoBleM

Some bots are used to launch attacks, send spam, distribute malware, or just await further instructions. Often, some machines are infected by different types of malware, are controlled by different bot owners, and can be used in different attack campaigns.

While there are bot detection systems available in the market, these are often offered as add-on functionality to existing security tools, or as an integral function of a broader security objective, such as advanced persistent threat (APT) protection, malware detection, or DDoS mitigation. Attempting to manage bots by using add-on functionality included with related network security tools can provide some utility, but will not provide a complete bot management solution.

For example, solutions that focus solely on DDoS mitigation may block known DDoS bots, but allow continued harassment by Web scrapers and spam bots. Essentially, network security tools that focus on the end actions of a bot have not fully addressed the bot challenge, because the bot can be repurposed to carry out other actions.



Bot maNagEmENt aS aN EmErgINg markEt

Basic bot detection and controls are insufficient. Add-on or integrated bot detection technologies provide incremental improvements, but are ultimately incomplete. As a result, the emergence of solutions dedicated to the process of bot management is imminent.

defininG Bot ManaGeMent

As dedicated solutions for bot management begin to emerge in the marketplace, a clear definition of the capabilities of a bot management solution is a necessary first step.

First, bot management solutions should be able to distinguish bots from human end users. Existing bot detection tools spend a significant effort attempting to establish, with confidence, whether a particular device is a bot or a human user. They utilize micro-challenges such as tracking cookies, JavaScript injection, and mouse movement detection. More intrusive means such as the Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) tests should be optional and a last resort because of the disruption to the end-user experience. This first step is not trivial, as discussed in the August 2014 SPIE report.7

However, bot management solutions should go beyond simple bot detection to also provide analysis of bot actions, intentions, and history. Bots should then be categorized; a greater granularity in bot categories will provide businesses with more precise controls. Bot management solutions should also offer a simple process for defining their own bots. This is an important option because businesses use bots to interact with partner systems.

Importantly, the solution must also monitor and re-evaluate known bots, as their capabilities and intentions can and do change over time. Bots can become more offensive (going from light data collection to participating in DDoS attacks), or can become more acceptable (through clean-up and removal of offending bot programming). Continuous cloud security intelligence will help enable robust bot identification, classification, and tracking.

Finally, bot management solutions must provide a range of controls and response mechanisms. Most bots will not fall into a simple bad or good category; and, likewise, response options should expand well beyond binary “allow” or “deny” actions. Some bots provide services that are desired, but that are enacted in an overly aggressive manner, such as the Chinese search engine Baidu. In this case, businesses may want to direct bots to the robot.txt file, or perform some other type of throttling. Businesses may also want to provide differential treatment of bots that are specifically known to them, such as partners or contracted third-party services.



Therefore, these options should be available to manage relationships with specific bots, categories of bots, or custom bots. Exhibit 1 below provides a few bot response options that would benefit customers.

Exhibit 1: Bot Response Options

Bot Management Action Appropriate for These Bot Types

Allow the bot to perform its actions

Good bots such as Google search bots and partner bots offer very desirable services and should be allowed.

Block the botThis option is most applicable for malicious bots; however, there are better options for dealing with malicious bots.

Direct a bot to an alternate Web page

This option allows businesses to leverage misdirection as a means to deal with unwanted bot traffic. Misdirection prevents competitors or attackers from realizing that their bots are ineffective and prevents bot mutation.

Throttle trafficThis is an appropriate response for partner bots that are overly aggressive by establishing timeframes for communications that are acceptable to both parties.

Direct the bot to appropriate APIs

This is a useful option for partners. Application programming interfaces (APIs) provide more effective and streamlined controls for transacting with partner systems.

Source: Stratecast

Beyond these essential capabilities, bot management vendors can distinguish their solutions from the competition by providing user-friendly interfaces, simplified categorization, and fast implementation. Vendors can further meet customer needs by supporting both cloud-based and on-premises deployment options. This capability will provide protection for application workloads in cloud-based data centers such as Amazon AWS and Microsoft Azure.

UndeRstandinG Bots as a tool of hUMan aCtoRs

To manage bots more effectively requires the security industry to maintain an understanding of the people behind the tools. While the default response to an unwanted bot is to block it outright, this strategy is not recommended. Blocking a bot outright will cause the bot owners to modify their bots to avoid detection mechanisms. Essentially, bot blocking sends a feedback signal that alerts bot owners, thereby driving bot evolution. Instead, by directing a bot to a purpose-built server with alternate, falsified data such as fake inventories and incorrect prices (misinformation), the bot will report success, while delivering incorrect data to the bot owner.

seCURity solUtion aRChiteCtURes that sUppoRt Bot ManaGeMent

Vendors with backgrounds in WAF, CDN, and application delivery controllers (ADCs) are the primary candidates to offer complete bot management solutions. These solutions have excellent visibility and control of application-layer traffic.

For example, WAF solutions, when deployed as full reverse proxies, are able to inject micro challenges into Web traffic to determine whether a device is a bot or human. These solutions follow an escalation pattern, beginning first with less-intrusive methods, and then using more accurate but more invasive techniques such as CAPTCHA, as needed.

A CDN operator also has tremendous network visibility. CDNs deploy and manage servers across many different service provider networks around the world to route traffic and deliver content efficiently, which allows the CDN to accelerate Web transactions. As the inverse of this capability, CDN operators are able to block or limit unwanted traffic from their network edge.



A CDN operator maintains a high level view of the Internet threat landscape, which enables the company to identify attacks that might be vastly distributed or well-coordinated. These attacks resemble the feeding style of piranhas, with each bot taking a small bite of bandwidth before allowing the next bot to attack. By comparison, an on-premises WAF may not have the visibility necessary to identify such a large distributed and coordinated effort.

NEw aNd EmErgINg Bot maNagEmENt SolutIoNS

Customer interest in bot management solutions has already reached a substantial level and will only grow further in direct relation to increases in bot activity. As a result, security vendors are developing and releasing new solutions specifically designed to handle the bot challenge. In December 2014, Imperva introduced its ThreatRadar Bot Protection Service as an add-on capability for its SecureSphere WAF. The solution can distinguish bot from human activity, categorize bot types, and eliminate unwanted bot activity. Imperva WAF capabilities are also delivered through the Incapsula cloud-based solution, which also provides DDoS mitigation, load balancing, and CDN services.

The introduction of purpose-built bot management services with more comprehensive capabilities will drive market growth further. For example, Akamai introduced its Bot Manager product in February 2016 as a purpose-built bot management solution on its CDN platform. Akamai has thousands of CDN customers, many of which are leveraging its cloud security services. This bot management service will be one of the first solutions capable of managing the bot challenge in totality, with a wide range of bot management actions, as well as very granular visibility into changing bot traffic.

the iMpoRtanCe of CloUd intelliGenCe foR Bot ManaGeMent

Bot management is a never-ending game of cat and mouse. Bot management solutions can catch a bot, but then the bot mutates and evolves to once again evade detection. Therefore, bot management vendors need visibility into global Web traffic in order to see how bots are changing, so they can adapt their detection and management techniques, and stay ahead in the game.

For example, the Akamai Cloud Security Intelligence data analysis engine processes 2 Petabytes of security data, ingesting 20 Tbps of new security data daily. The vast amount of data that Akamai can see, from monitoring up to 30% of Internet traffic daily, includes logs, attack data, and network flows. This data is used to identify and track over 1,300 known bots in 15 categories of legitimate Web and business services, and millions of unique IP addresses performing Web scraping activities every month, as well as to continuously refine its real-time bot detection rules to keep up with evolving unknown bots.



StratEcaSt: thE laSt word

The unique challenges presented by automated bots necessitate the development of new solutions. Dedicated bot management solutions can enable businesses to reduce their risk significantly by disabling the tools that hackers use to launch DDoS attacks, intrusion attempts, and to distribute spam and malware. Bot management solutions will provide the functionality necessary to control, limit, or evade bots that operate in multiple shades of legal or ethical gray areas, such as data collection and Web scraping.

Bot management provides benefits that go well beyond risk mitigation, including IT and business process optimization. Precision detection and management of bot activity can provide tangible business and operational benefits by prioritizing Web indexing, SEO, and partner bots, while throttling and limiting overly enthusiastic bots. Businesses will be able to reduce unwanted Web traffic, reduce networking costs, improve search engine rankings, and acquire more accurate analytics. Interestingly, the use of advanced security analytics and big data to identify and track bots may even enable the possibility of “predictive” security on a long enough time scale.

Demand for bot management solutions is steadily mounting. As a result, the emergence of dedicated bot management solutions is imminent. Imperva and Akamai both offer examples of emerging bot management services that provide key elements of a dedicated bot management solution, as outlined in this report, and provide a framework for future bot management solutions.

Chris Rodriguez sr. industry analyst – information & network security frost & sullivan [email protected]

frost.com



E N D NOT E S

1. Please note that the insights and opinions expressed in this assessment are those of Stratecast and have been developed through the Stratecast research and analysis process. These expressed insights and opinions do not necessarily reflect the views of the company executives interviewed.

2. SPIE 14-32, The Forgotten Barometer: Bot Detection as an Integral Security Technology, August 29, 2014. Available at http://www.frost.com/q292040677.

3. https://blogs.akamai.com/2016/02/all-we-are-saying-is-give-bots-a-chance.html

4. Bot and Spider Filtering, Google, July 2014. Available at https://plus.google.com/+GoogleAnalytics/posts/2tJ79CkfnZk.

5. International IAB/ABC Spiders & Robots List, Audit Bureau of Circulations (ABC). Available at: http://www.abc.org.uk/Products-Services/Spiders--Bots-/

6. 2015 Bot Traffic Report, Imperva, December 2015. Available at: http://lp.incapsula.com/HumansTakeBacktheWeb_LP.html

7. SPIE 14-32, The Forgotten Barometer: Bot Detection as an Integral Security Technology, August 29, 2014. Available at http://www.frost.com/q292040677.

[email protected]

SILICON VALLEY331 E. Evelyn Ave., Suite 100

Mountain View, CA 94041

Tel 650.475.4500

Fax 650.475.1570

SAN ANTONIO

7550 West Interstate 10,

Suite 400

San Antonio, TX 78229

Tel 210.348.1000

Fax 210.348.1003

LONDON4 Grosvenor Gardens

London SW1W 0DH

Tel +44 (0)20 7343 8383

Fax +44 (0)20 7730 3343

AucklandBahrainBangkokBeijingBengaluruBuenos AiresCape TownChennaiDammamDelhiDetroitDubaiFrankfurtHerzliyaHoustonIrvineIskander Malaysia/Johor BahruIstanbulJakartaKolkataKotte ColomboKuala LumpurLondonManhattan

MiamiMilanMoscowMountain ViewMumbaiOxfordParisPuneRockville CentreSan AntonioSão PauloSeoulShanghaiShenzhenSingaporeSydneyTaipeiTokyoTorontoValbonneWarsaw

Frost & Sullivan, the Growth Partnership Company, works in collaboration with clients to leverage visionary innovation that addresses the global challenges and related growth opportunities that will make or break today’s market participants. For more than 50 years, we have been developing growth strategies for the Global 1000, emerging businesses, the public sector and the investment community. Is your organization prepared for the next profound wave of industry convergence, disruptive technologies, increasing competitive intensity, Mega Trends, breakthrough best practices, changing customer dynamics and emerging economies?

For information regarding permission, write: Frost & Sullivan 331 E. Evelyn Ave., Suite 100 Mountain View, CA 94041

bots to the future - frost & sullivan · 2016-03-22 · distracting, and unrewarding. many...

Documents