Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com

February 2015 $99

Building a Security Analytics Initiative To identify sophisticated attacks, infosec teams must correlate a huge

range of data — from internal systems, threat intelligence services, cloud

and network service providers, digital forensics and attribution services,

and others. One way to cope: big data tools and practices.

By Michael Cobb

Report ID: S8320115

reports

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

CONT

ENTS

TABLE OF

reports.informationweek.com February 2015 2

3 Author’s Bio

4 Executive Summary

5 Harder Than It Looks

5 Big Tools, Big Plans

5 Figure 1: Data Analysis Goals

6 Figure 2: Factors Driving Interest

in Big Data Analysis

7 Choosing the Right Strategy

8 Getting the Best Out of Big Data

8 Figure 3: Build vs. Buy

9 Security vs. Privacy

10 Can Big Data Improve Security?

10 Figure 4: Value of Threat Intelligence

11 Figure 5: Analysis of Internally

Generated Data

12 Related Reports

reports

ABOUT US

InformationWeek Reports’ analysts arm business technology decision-makers with real-world perspective based on qualitative and quantitative research, business and technology assessment and planning tools, and adoption of best practices gleaned from experience. OUR STAFFLorna Garey, content director; lorna.garey@ubm.comHeather Vallis, managing editor, research; heather.vallis@ubm.com Find all of our reports at reports.informationweek.com.

Building a Security Analytics Initiative

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com February 2015 3

Table of Contents

Michael CobbInformationWeek Reports

Michael Cobb, CISSP-ISSAP, is a renowned security author with over 20 years of experi-ence in the IT industry. He co-authored the book IIS Security and has written numerous technical articles for leading IT publications. He was also formerly a Microsoft Certified Database Manager and a registered consultant with the CESG Listed Advisor Scheme (CLAS). Mike has a passion for making IT security best practices easier to understand and achievable. His website, The Hairy IT Dog, offers free security posters to raise employee awareness of the importance of safeguarding company and client data and of following good practices.

© 2015 InformationWeek, Reproduction Prohibited

reports Building a Security Analytics Initiative

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com February 2015 4

Table of Contents

SUM

MAR

Y Many security teams empathize with Italian artist Sven Sachsalber, who recently spent 48 hours in a Paris museum looking for a needle in a haystack. At least Sachsalber knew what he was seeking, and the haystack didn’t keep increasing in size every hour. Those responsible for network security should be so lucky.

In the face of huge and diverse stores of data, many enterprises are turning to big data analytics to help find threats faster and with more accuracy. Threat detection methods based solely on known elements, such as whitelists and blacklists, signature lists, and rule lists, aren’t effective against the unknown. However, searching through vast amounts of data can unearth clues that make anomaly detection techniques more effective in spot-ting malicious activity, while behavioral analytics can better distinguish between legiti-mate and suspicious users. Gartner predicts that by 2016 more than 25% of global firms will adopt big data analytics for security and fraud detection.

In this report, we discuss tools and methods for harnessing all available security informa-tion and for building a forensic analysis process that can lead to faster identification of targeted attacks and to better strategies for enterprise data defense.

EXECUTIVE

reports Building a Security Analytics Initiative

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

Building a Security Analytics Initiative

reports.informationweek.com February 2015 5

Table of Contents

Harder Than It LooksThe marketing hype that surrounds big data analysis for improved security makes it all sound so easy. In reality, big data technologies are still evolving and are by no means plug-and-play. There are plenty of pitfalls you need to avoid if you want to make the investment, time, and ef-fort yield real results. Turning big data into threat intelligence requires specialists armed with the right tools to uncover hidden patterns and un-known correlations that can help identify a data breach or an ongoing attack. Project schedules and budgets need to take into account custom development work and the fact that everyone will be on a steep learning curve.

The complexity of big data is what makes it different from more conventional data when it comes to storage and query management. Technologies designed specifically to handle semi-structured and unstructured data that don’t fit into traditional data warehouses are essential, as is an infrastructure designed to handle the processing demands posed by sets of big data that need to be continually updated. However, I advise against abandoning existing data warehouses and business intelligence and

data visualization tools — instead, choose big data systems that work with and complement existing technologies.

Big Tools, Big PlansWith so many organizations moving to take

advantage of big data in all areas of business, it’s no wonder we’re seeing a flood of new tools and technologies for housing and analyzing

large-scale, heterogeneous datasets at extraor-dinary scales and speeds. Most are designed to be run as part of a Hadoop system, an open-source software framework inspired by Google’s MapReduce, initially developed by Yahoo to store and process the ever-expanding number of web pages and searches it was struggling to handle. It is now an Apache Software Founda-tion project and at the heart of what’s known as

Figure 1

reports

What are you typically attempting to identify through your data analysis?

Data Analysis Goals

Vulnerabilities exploited

Assets under attack

Source of the attack

Failed defenses

Method of the attack

Other

Note: Multiple responses allowed Base: 337 respondents using threat intelligence Data: Dark Reading Threat Intelligence Survey of 397 business technology and security professionals, June 2014

R8040914/11

R8040914/11_Threat Intelligence

75%

74%

58%

56%

51%

5%

reports

25%of Dark Reading Threat

Intelligence Survey

respondents built their own

security analytics systems.

FAST FACT

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com February 2015 6

Table of Contents

the Hadoop Ecosystem, an array of related proj-ects that work on top of or alongside Hadoop, such as Apache Hive (a SQL-friendly query lan-guage), Apache Pig (a platform and a scripting language for complex queries), HBase (a non-relational, distributed database modeled after Google’s BigTable), and Zookeeper (a central-ized service for highly reliable distributed co-ordination). Each project has been developed to deliver an explicit function, and each has its own community of developers and individual release cycles.

In addition to these projects, a lot of startups are building higher-level applications, frame-works, and management software around Hadoop. Most are aiming to deliver faster and easier querying of ever larger datasets, while others are concentrating on making the entire infrastructure more manageable. Before choos-ing any set of technologies, it’s important to research the language in which they’re written, who supports their development, and whether they can deliver the key features you need.

For example, there are several NoSQL data-bases designed specifically for storing and que-rying big data, such as Cassandra, VoltDB , and

MongoDB. Some are written in Erlang , which may not be a skill set you have in-house. If that’s the case, those written in Java or C++ may be a better choice.

The strength of the developer community or company behind a specific tool is key to its success and longevity, but in such a new and fast-moving industry, life expectancy can’t be

reports Building a Security Analytics Initiative

2015 2014

What data sources or challenges are driving, or would drive, your organization’s interest in doing big data analysis?

Factors Driving Interest in Big Data Analysis

Finding correlations across multiple, disparate data sources (clickstreams, geospatial, transactions, etc.)

Predicting customer behavior

Predicting product or service sales

Predicting fraud or financial risk

Analyzing social network comments for consumer sentiment

Note: Multiple responses allowedBase: 297 respondents in October 2014 and 248 in October 2013 at organizations using or planning to deploy data analytics, BI, or statistical analysis software Data: InformationWeek Analytics, Business Intelligence, and Information Management Survey of business technology professionals

44%46%

43%48%

36%40%

32%27%

29%24%

29%25%

28%23%

24%26%

1%3%

12%14%

R8241114/10

Analyzing high-scale machine data from sensors, web logs, etc.

Identifying computer security risks

Analyzing web clickstreams

Other

Big data analytics is not of interest to my organization

Figure 2

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com February 2015 7

Table of Contents

guaranteed. For example, the community edi-tion of the Greenplum database has been discontinued since becoming part of Pivotal Software’s Big Data Suite. While most big data software is generally open source, be aware that enterprise-level support usually isn’t.

Various tools can help analysts create complex queries and run machine-learning algorithms on top of Hadoop, but some, like Mahout, are aimed more at the needs of the marketing and sales department than a security team. Drill, in-spired by Google’s Dremel, and Spark are better choices and improve the efficiency of data min-ing and machine learning algorithms. Splunk’s Hunk for Hadoop and NoSQL data stores is another strong contender, given Splunk’s back-ground in security. In fact, Symantec has started using Splunk Enterprise to centralize, monitor, and analyze its security-related data.

Choosing the Right StrategyBig data analytics is a fast-moving discipline.

There are on-premises, cloud, and hybrid sys-tems to choose from, but most options are far from mature. The 2014 Wikibon Big Data Ana-lytics Survey found that the average existing

big data analytics project returns just 55 cents for every dollar invested. This shows that prop-er research and planning are needed to avoid disappointment, and the evaluation and selec-tion of big data technologies should not be left entirely to the security team. Get input from network architects and trained data analysts. InformationWeek’s 2015 Analytics, Business Intelligence and Information Management Survey provides insights into challenges and opportunities.

Agile development methodologies are a bet-ter fit for building a security analytics initiative based on big data than is a standard waterfall approach. An agile approach delivers functional-ity in small, iterative chunks and accommodates quick changes in development plans, which are inevitable in this type of venture.

For those new to big data, one way to reduce cost and risk while accelerating deployment is to go for an integrated environment: Create a data refinery to handle the “volume, variety, velocity, and veracity” of big data and to stan-dardize it so that it can be fed into existing data warehouse applications and appliances for analysis. This is mainly a batch-processing solu-

tion for analyzing data at rest.For those requiring near-real-time results,

consider a central big data repository for all incoming streams of raw data. The idea is to analyze data directly in the repository, possibly using in-memory analytics tools — though the cost of RAM can make this an expensive op-tion. Smaller subsets can still be created and offloaded for analysis by standard business in-telligence software.

Integrating Hadoop systems and data ware-houses can be a challenge, although various vendors now offer software connectors be-tween Hadoop and relational databases, as well as other data integration tools with big data ca-pabilities. Cloud-based analysis is an option us-ing the likes of Google’s BigQuery data analytics service and IBM’s Bluemix cloud platform. While it’s less expensive to run cloud-based machines than buy real ones, it may not be suitable for those who need a secure, stable, and auditable environment or where the cost of moving data to a private cloud is prohibitive.

For those who don’t feel ready to tackle such a bewildering and confusing set of choices, various well-known names offer supported big

reports Building a Security Analytics Initiative

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com February 2015 8

Table of Contents

data packages and platforms. IBM has launched a number of cloud-based and on-premises big data and analytics tools, and Microsoft and Tera-data offer software and first-line support for the Hortonworks platform. Oracle resells and sup-ports Cloudera, while HP and SAP work with various Hadoop software providers.

This is one competitive market, so make the most of trial periods. Really test the capabilities of different offerings and the levels of support provided before committing.

Senior management needs to be aware of the pros and cons of the technologies you’re considering so that expectations remain realis-tic. One factor that has to be taken into account when selecting big data technology is the avail-ability of people with the necessary skill sets to make it all work. With the surge in interest in big data there is huge demand for people who have experience in unstructured data systems like Hadoop and NoSQL and other related skills such as Python, a popular choice for building data processing scripts.

Getting the Best Out of Big DataBig data systems running on powerful, high-

ly distributed architectures can be a game changer in the hunt for security breaches, while helping you understanding how to improve network defenses. People with very specific skills are needed, though, to use big data ef-fectively, and these specialists are in short sup-ply, particularly when it comes to some of the newer technologies. System architects and ad-

ministrators are needed to implement, config-ure, optimize, manage, and secure a distributed architecture. Interacting with the data requires a detailed knowledge of a variety of program-ming languages to write custom integration code.

Those in an analysis role will need ex-perience in working with structured and

reports Building a Security Analytics Initiative

Did your organization build its own security analytics system or use a third-party offering?

25%

7%

42%

26%

Build vs. Buy

Base: 337 respondents using threat intelligence Data: Dark Reading Threat Intelligence Survey of 397 business technology and security professionals, June 2014

R8040914/17

R8040914/17_Threat Intelligence

A combination of both

Used a third party to understand how to build our own customized system

Self-build

Third party

Figure 3

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com February 2015 9

Table of Contents

unstructured data using techniques such as multivariate analysis, predictive modelling, natural language processing, and content analysis. There are no restrictions on the data that the Hadoop Distributed File System can store, but to run more sophisticated detec-tion algorithms than a traditional SQL-based data system requires mastering various SQL-like languages. Tools like HP’s Vertica for SQL on Hadoop aim to simplify querying large amounts of data. Companies such as Clou-dera, MapR, and Hortonworks offer training courses in Hadoop. Still, many organizations will have to turn to external partners for the necessary skills.

Generating insights into malicious network activity from big data is an iterative process. Organizations that succeed in using big data for security encourage data exploration. Is there another variable out there that can pre-dict an attack? Tools with good visual analyt-ics will not only help analysts interpret query results, but also make conveying the informa-tion to other stakeholders a lot easier. Big data allows analysis based not only on large data sets, but also on large numbers of attributes

per record, and that increases predictability. The cross-correlation of data from multiple devices can improve the accuracy of threat identification, while effectively joining secu-rity devices together that must be defeated as a whole, rather than serially.

The threat intelligence that comes from ana-lyzing an attack’s source, delivery system, and payload can be used to search for and block similar attacks, and to analyze live traffic for re-lated attack fingerprints. Researching historical data is also important to find when and how a breach may have occurred, and what the con-sequences were. Clear reports on any forensic evidence uncovered can help reduce the cost and workload of post-attack recovery.

Data analysts will need to think like hackers to avoid the trap of mirror imaging, where an-alysts imagine that the enemy thinks as they do. Another trap is target fixation, where an analyst becomes fixated on one hypothesis and only sees the evidence that supports it. Putting analysts through a pen-test training course and ensuring they stay abreast of new attack techniques can help overcome these problems.

Security vs. PrivacyThe big data infrastructure you build needs

to be highly protected, since it’s an obvious target for would-be attackers. However, the non-scalability of encryption for large data as-sets and the use of multiple infrastructure tiers make security a real challenge. Non-relational databases are only slowly beginning to add tra-ditional enterprise database security features such as access control. These databases were initially developed to tackle specific analytical challenges, and so security was never made a priority. For example, Google invented MapRe-duce to store public hyperlinks — data with no real security or privacy requirements. The Cloud Security Alliance’s Top Ten Big Data Security Challenges report highlights infrastructure se-curity and data privacy as two of the four main challenges facing big data ecosystems. Syman-tec’s NetBackup Data Protection Platform is being used by various enterprises to integrate data management and protection into their big data solutions.

A big data project of any kind needs a frame-work in place for preventing inadvertent pri-vacy disclosures in order to ensure the privacy

reports Building a Security Analytics Initiative

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com February 2015 10

Table of Contents

principle of avoiding data reuse is not violat-ed as you pursue those behind malware and attacks. Data can be used only for the pur-poses for which it was collected, so adding emails, tweets, and blog posts to log files for analysis is possible only if users are aware that this will happen. This is still somewhat of a gray area, and the industry is still looking for the best approach to balance the need for privacy and the need to know exactly what is happening on the network. A review of pri-vacy policies and agreements is essential to ensure you stay on the right side of regula-tory rules and regulations.

Can Big Data Improve Security?Enlisting big data analytics in the war

against today’s sophisticated attacks is a smart move and a great strategy to improve detec-tion rates. The capacity to see the big picture at all times makes analysis far more likely to, for example, expose the orchestration of an APT-style attack. The statistical analysis of so much data can help security professionals de-termine a baseline of what is normal, so that abnormal activity can be quickly detected and

used as a starting point for further investiga-tion. By leveraging machine learning, big data analytics can also enhance intrusion detection and intrusion prevention systems by deliver-ing continuous fine tuning.

Big data helps overcome the inherent weak-ness of static, signature-based tools. Math-based anomaly detection does not have to

know what a threat might look like. It simply looks for irregularities and anomalous behav-ior. This form of threat detection can prevent breaches or detect malicious behavior once a breach has occurred and can minimize its im-pact. Information can be used not only to stop and close out an attacker, but to adjust secu-rity strategies and defenses to stop similar or

Threat Intelligence TodayIs threat intel the best way to stay ahead of new and com-plex attacks? The 397 respon-dents to our new survey seem to think so.

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports

Is threat intelligence plus the tools needed to analyze it a worthwhile investment?

13%

61%

12%14%

Value of Threat Intelligence

Base: 337 respondents using threat intelligence Data: Dark Reading Threat Intelligence Survey of 397 business technology and security professionals, June 2014

R8040914/16

R8040914/16_Threat Intelligence

It’s good value for the money

Expensive for what we get out of it

Other security tools are a better value

It’s worth its weight in gold

Figure 4

Building a Security Analytics Initiative

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com February 2015 11

Table of Contents

related attacks in the future. Note that having a security analytics program

doesn’t mean that point defense technologies, like firewalls and antivirus, are no longer rele-vant: Not only do point systems provide a layer of defense, analyzing what these devices are blocking can provide insights into areas that need deeper investigation.

Although threat management begins with threat identification, remediation is an es-sential next step. Analytical tools will create more value if their outputs are incorporated into incident management processes so re-sponse and remediation can start as soon as possible. Once a threat is identified, the right people need to be alerted and automated re-sponses triggered — think scripts to update security devices.

Building a security analytics system, espe-cially one involving big data, is complex and resource-intensive. Budget time to design the architecture and then choose the technology elements best suited to manage and analyze your data. And, don’t underestimate demand for expertise. People working with well-de-

signed analytical processes who have the tal-ent and skills needed to leverage these new technologies are essential. Many big data tools are open source, so it can be tempting to just go out and load up on software. Introducing any new technology is a challenge, and having to hire programmers with skills in each set of

tools will add unnecessary costs. Finally, consider sharing the threat intelli-

gence and knowledge of attack data gained through your research with a relevant threat-sharing community, since community defense is a great example of how security analytics can really make a difference.

Like This Report?

Share it!FollowFollow

Follow

TweetTweet

LikeLike

Follow

ShareShare

Next

FollowFollow

Follow

TweetTweet

LikeLike

Follow

ShareShare

Next

FollowFollow

Follow

TweetTweet

LikeLike

Follow

ShareShare

Next

reports Building a Security Analytics Initiative

Have you fully realized the benefits of analyzing internally generated data?

44%

24%

7%4%5%

16%

Analysis of Internally Generated Data

Base: 337 respondents using threat intelligence Data: Dark Reading Threat Intelligence Survey of 397 business technology and security professionals, June 2014

R8040914/6

R8040914/6_Threat Intelligence

Not yet, but we’re getting there

Yes; we’re pretty much there

Don’t know

No, nowhere close

Yes, to the extent we need additional third-party feeds

Yes, but we can’t process more data than we already have

Figure 5

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com February 2015 12

Table of Contents

MOR

ELIKE THIS

Want More Like This?

InformationWeek creates more than 150 reports like this each year, and they’re all free to registered users. We’ll help you sort through vendor claims, justify IT projects, and implement new systems by provid-ing analysis and advice from IT professionals. Right now on our site you’ll find: Threat Intelligence Today: Is threat intel the best way to stay ahead of new and complex attacks? The 397 respondents to our new survey seem to think so: 85% say threat intelligence plays some role in their IT security strategies, and many of them subscribe to two or more third-party feeds; 10% leverage five or more.

Making Threat Intelligence Services Work for Your Enterprise: The more IT protects networks and data, the more attackers will look elsewhere for weak links. Don’t underestimate your ability to demand that contractors and suppliers shore up their defenses. 2015 Analytics & BI Survey: A new spirit of experimentation is in the air: For the first time in years, more companies are trying out multiple anaytics tools rather than standardizing on a few favored ones. Data qual-ity and ease of use remain the toughest challenges. The key question: Will buyers look for simpler or smarter tools for big data analytics?

PLUS: Find signature reports, such as the InformationWeek Salary Survey, InformationWeek Elite 100, and the annual State of Security report; full issues; and much more.

reports Building a Security Analytics Initiative