my keynote from bsidestampa 2015 (video in description)
TRANSCRIPT
The Need for Proactive Threat Hunting
Andrew Case
@attrc
Who Am I?
• Core Volatility developer
• Co-Author “Art of Memory Forensics”
• Lead-investigator on large-scale investigations
• Performed many RE efforts, pentests, and source code audits
• BSidesNola (New Orleans) Co-Organizer
Why is Threat Hunting Needed?
• Many opaque components of the information infrastructure
• You are combating a creative and adaptive adversary and thus you need a creative and adaptive analyst to find them
• Statistics have shown that people are compromised for years without noticing
What is Threat Hunting?
• Searching for adversaries without a particular indicator
• Dedicating time and resources to deep analysis of potentially compromised resources
• See [1] for great commentary by Sean Mason and [2] for several posts by Jack Crook
What are its Benefits?
• Makes the organization proactive against attackers
• Quickly find gaps in system and application configurations
• Defenders more familiar with their own environment and infrastructure
• Documentation leads to organizational knowledge
Gaining Familiarity
• Understanding and defining “normal” in order to detect anomalous behavior and attributes
• “normal” is unique to a particular organization and even subsets within the organization
– “normal” of a web server is quite different than the system of Joe in accounting
• Unfamiliarity with “normal” leads to extremely ineffective response
Running Processes
• If your analysts were given a list of every process running on a system in your environment, how many of them could definitively rule each as normal or abnormal?
• How would this be judged?– Name of the process?
– Path to the executable on disk?
– Parent process?
• Patrick Olsen has gone through great lengths to document this [5]
Process Privileges
• What privileges do each process run as?
• Do any 3rd party programs abuse privileges or grant themselves higher privileges than necessary?
• Do you know which of your users run as local admin?
Network Activity
• Which applications should be listening for network connections?
• Which applications should talk on the network?
• Is there any ingress/egress filtering?
– Has it been disabled or tampered with by malware/attackers?
Kernel Drivers
• Kernel drivers have full access to entirety of a system and its resources
• A default Windows 7 install loads over 100 kernel drivers
• Two of the following drivers are normal, two are Stuxnet, do your analysts know which?– MRxCls
– MRxDAV
– MRxNet
– MRxSMB
Persistence Mechanisms
• More than just AutoRun Keys
• “Beyond the Run Key”, 26 (now 27) part and counting blog post series by Adam Blaszczykon forensics of Windows persistence mechanisms [4]
Scheduled Tasks
• Default Windows 7 install has numerous scheduled tasks by default
• 3rd party applications create them to check for updates, run maintenance scripts, and so on
• Adversaries also leave scheduled tasks behind…
• Triggers: [6]
Services
• Like Scheduled Tasks, many will be installed by default and 3rd party applications will create their own
• Sophisticated threat groups also like to install services
– Can be very manual and time consuming to detect malicious services, even with memory forensics analysis
Anti-Virus & HIPs
• Often act like malware to gain visibility into the system
• Need to be filtered/whitelisted from any rootkit detection tools
• Changes in AV/HIPs algorithms requires changes in filters
Documentation is Org Knowledge
• Team members should not live in a silo
– “normal” should be documented in a way that other team members can access
• Documentation outlives employees leaving and scales during incidents
– Do not allow “Brents” to be created
• If your entire IR team mutinied tomorrow, how long would it take for new hires to regain all the departing knowledge?
What is the End Result?
• Proactive detection of threats
• Effective detection and response
• IR teams that deeply understand their environment
• Organizational knowledge that continues to grow and survives generations of employees
How Do You Get There?
• The executives need to understand the value of a properly prepared IR team
• The IR team must be elevated to the status of the IT Security team and be just as an integral a part of the organization’s ongoing IT flow
Security vs IR
• Security teams are positioned during all parts of the IT process while IR is used only during incidents
• This leads to IR staff not being effectively utilized and not being an on-going part of the organization
IT Security Pre-Deployment
• Baseline testing of gold images
– Security evaluations done well before production use
• Application development
– Secure SDLC
• Secure DevOps
– Incorporating security into cloud deployments
– Richard Mogull does great work in this space [3]
IT Security Post-Deployment
• Continuous:
– Vulnerability scans
– Penetration tests
– Application security assessments
IR is Embedded Into Nothing
• It is always after the fact
• This leaves knowledge gaps and forces on-the-spot learning during incidents
• How do we fix this?
Incorporating the IR Team Pre-Deployment
• As security reviews gold images, the IR team should be building baselines and looking for logging misconfigurations that prevent full forensic exploitation
• Applications should be developed and configured so that all relevant activity is logged and recoverable
Incorporating the IR Team Post-Deployment
• Continuous:
– Threat hunting
– Documentation of changes to systems and applications
– Incorporation of new forensics artifacts into analysis processes
Incident Preparedness
• IT security has dedicated systems for vulnerability scanning, application testing, etc.
• IR teams need dedicated, pre-configured systems to effectively hunt as well as respond to incidents
Incident Preparedness Essentials
• Network monitoring
• Dedicated storage servers
• Deployable acquisition/sampling tools and agents
• Analysis servers with real processing power
• Without these and others, response will be chaotic, underpowered, and likely ineffective
Utilizing Documentation
• As the IR team becomes embedded, everything it learns should be documented
• If done correctly, everything that is known from a forensics perspective about a system and its applications will be readily available to all team members
Documentation into Internal Training
• New hires can be pointed to documentation of all assets on the network and their forensic value and artifacts
• Ongoing internal training can focus on new artifacts discovered during all phases of the IR team’s involvement
• A great post by Jack Crook that covers this topic [7]
Helping Outside Parties
• Only a handful of organizations can completely handle major breaches internally
• Giving organized access to 3rd party analysts makes their effort more effective and accomplished in a shorter amount of time
• To accomplish this, documentation and analysis infrastructure must be setup before a breach
About Executive Support…
Spending: Security vs IR Preparedness
• If “Shell Shock 2” were to be released right now would you feel better knowing your systems were fully patched (hence vulnerable) or that you had a fully prepared IR team that can handle the outbreak effectively?
• Does your organization’s resource allocation reflect your feelings on this?
Steps – Preparing for a Hunt
1. Free up your IR teams time to prepare for hunting
2. Create a plan that will lead to documentation of all your hunting and response efforts
3. Start small
4. Refine
5. Move to team wide hunts
6. Create challenges and internal training based on real events
Steps – Embedding IR into IT
1. Convince executives of the need
2. Update policy to ensure IR has a hand in ongoing operations just like security does
3. Document everything learned
4. Incorporate what is learned into analysis during hunts and incident handling
Conclusions
• Threat hunting is one of the best tools available to organizations in order to stay ahead of adversaries
• You should aim to minimize the space attackers can work where you will not find them
• Don’t wait on a vendor or the FBI to notify you of breaches – be active and find them yourself!
References
[1] http://seanmason.com/2014/12/09/a-hunting-we-will-go/
[2] http://blog.handlerdiaries.com/?s=hunting&submit=Search
[3] http://2014.video.sector.ca/video/110341603
[4] http://www.hexacorn.com/blog/
[5] https://sysforensics.org/2014/01/know-your-windows-processes.html
[6] https://technet.microsoft.com/en-us/library/cc748841.aspx
[7] http://blog.handlerdiaries.com/?p=437