hit ‘em where it hurts: a live security exercise on cyber situational awareness
DESCRIPTION
Talk I gave at ACSAC 2011 on the paper: "Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness" which describes the 2010 international Capture the Flag (iCTF) competition.Paper is located here:http://cs.ucsb.edu/~adoupe/static/hit-em-where-it-hurts-acsac2011.pdfTRANSCRIPT
![Page 1: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/1.jpg)
Hit ‘em Where it Hurts:A Live Security Exercise on
Cyber Situational Awareness
Adam Doupé, Manuel Egele, Benjamin Caillat, Gianluca Stringhini, Gorkem Yakin, Ali Zand, Ludovico
Cavedon, and Giovanni Vigna
University of California, Santa Barbara
ACSAC 2011 – 7/12/11
![Page 2: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/2.jpg)
What Are Live Security Competitions?
• AKA Hacking Competitions• Useful educational tool for teaching computer security• Born as a way to showcase security skills
– DefCon’s CTF
• Various forms– Challenge set (DefCon quals, iCTF challenges, CMU’s
competition, DIMVA competition, RuCTF)– Capture the flag (DefCon, iCTF 2003-2007, CIPHER)– Other designs
• Attack-only (e.g., iCTF 2008)• Defense-only (e.g., Cyber Defense eXercise)
![Page 3: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/3.jpg)
Doupé - 7/12/11
Why Live Security Competitions?
• Real-time factor enhances understanding• Forces teams to:
– Analyze unknown services/binaries– Defend systems from attack– Utilize different security skills– Work as a team– Create novel tools
![Page 4: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/4.jpg)
Doupé - 7/12/11
Key Insight
• Security competitions can be designed to generate datasets for research
• In the 2010 international Capture The Flag (iCTF), we structured the competition to create a Cyber Situational Awareness dataset
![Page 5: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/5.jpg)
Doupé - 7/12/11
Situational Awareness
• By putting perceived events into the context of the currently executing mission, one can improve decision making
• Mission– Series of tasks that an organization wishes to carry
out
• Task– Discrete step that is carried out using a service
• Service– Provided to users to accomplish a task
![Page 6: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/6.jpg)
Doupé - 7/12/11
Cyber Situational Awareness
• Situational awareness extended to the cyber domain
• Large organizations constantly under attack– Which attacks are important?– Which assets are important?
• “What if” scenarios
![Page 7: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/7.jpg)
Doupé - 7/12/11
Overview
• Live Security Competitions• Situational Awareness• Design of the 2010 iCTF• Cyber Situational Awareness Metrics• Lessons Learned• Conclusion
![Page 8: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/8.jpg)
The 2010 iCTF: A Cyber SA Competition
• Introduced the concept of cyber-mission• “Not all attacks are created equal”• Participants must be aware of cyber-
missions and cyber-assets• Attackers must time their attacks to cause
the maximum amount of damage
![Page 9: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/9.jpg)
The Setting
• Teams are part of a coalition to bring down the rogue nation of Litya
• LityaLeaks site used to leak description of Litya’s cyber-missions
• Litya’s network protected by a firewall and an IDS– If an attack is detected, nation’s access is shut off– Nations can bribe network administrator
• Litya has a botnet in each nation, stealing their money– If botnet is disabled, nation’s access shut off
• Money made by solving side challenges.
![Page 10: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/10.jpg)
CARGODSTR-TQ-1442
![Page 11: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/11.jpg)
COMSAT-WK-1127
![Page 12: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/12.jpg)
SEDAFER-GOT-BKT-8217
![Page 13: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/13.jpg)
DRIVEBY-DEPLOY-QFK-9751
![Page 14: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/14.jpg)
Doupé - 7/12/11
Petri-net Representation of Mission
![Page 15: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/15.jpg)
Service 1 Service 2 … Service 10
.. .
.
![Page 16: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/16.jpg)
Service 1 Service 2 … Service 10
The Bank
ScoreBot
.. .
.
![Page 17: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/17.jpg)
InternalNetwork
…
VPN server
Botnet C&C
The Bank
Service 1 Service 2 … Service 10ScoreBot
.. .
.
![Page 18: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/18.jpg)
InternalNetwork
…
VPN server
Firewall/IDS
Botnet C&C
The Bank
Briber
Flag Submission
Service 1 Service 2 … Service 10ScoreBot
.. .
.
![Page 19: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/19.jpg)
InternalNetwork
…
VPN server
Firewall/IDS
Botnet C&C
The Bank
Briber
Flag Submission
Service 1 Service 2 … Service 10ScoreBot
.. .
.
Challenges
ScoreBoard
LityaLeaks
![Page 20: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/20.jpg)
Service 1
InternalNetwork
…
VPN server
Firewall/IDS
Service 2 … Service 10 Botnet C&C
The Bank
ScoreBot
Briber
Flag Submission
.. .
.
Challenges
ScoreBoard
LityaLeaks
![Page 21: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/21.jpg)
Doupé - 7/12/11
Competition Overview
• December 3rd 2010 ~8 hours• 72 teams• ~900 participants (largest at the time) • 7 of 10 services compromised• 39 teams submitted 872 flags• 69 of 72 teams solved at least 1 challenge• 37 GB of traffic
![Page 22: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/22.jpg)
Doupé - 7/12/11
Analysis of iCTF Data
• Use the data to validate models and theories
• We introduce two Situational Awareness metrics:– Toxicity
• Capture the amount of damage an attacker has caused
– Effectiveness• Capture how effective the attacker was at causing
damage
![Page 23: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/23.jpg)
Analysis – CAD - Criticality
• C(s, t): service criticality [0,1]– Expresses the criticality of service s at time t– Function can have any shape
• iCTF: 1 when service active, 0 otherwise
Service: MostWanted
![Page 24: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/24.jpg)
Analysis – CAD - Attacker
• A(a, s, t): attacker activity [0, 1]– Represent the attacker’s activity with respect
to a service– Can have any shape
• iCTF: 1 when team attacked a service, 0 if no attack
Team: PPP Service: MostWanted
![Page 25: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/25.jpg)
Analysis – CAD - Damage
• D(s, t): Damage to the attacker [0, 1] – Represents the penalty for performing an
attack against service s at time t– Can have any shape
• iCTF: 1 when service is inactive, 0 when active
Service: MostWanted
![Page 26: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/26.jpg)
Analysis – Toxicity
![Page 27: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/27.jpg)
Analysis – Effectiveness
![Page 28: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/28.jpg)
Analysis – Toxicity of PPP
Team: PPP Service: OvertCovert
![Page 29: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/29.jpg)
Analysis – Toxicity and Effectiveness
![Page 30: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/30.jpg)
Doupé - 7/12/11
Overview
• Live Security Competitions• Situational Awareness• Design of the 2010 iCTF• Cyber Situational Awareness Metrics• Lessons Learned• Conclusion
![Page 31: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/31.jpg)
Doupé - 7/12/11
Lessons Learned
• The Good– Pre-competition information prepared teams who
took advantage– Winning team automatically qualified for DefCon
• The Bad– Structure of the competition was complex and was
understood by a subset of the teams– Services too hard
• The Ugly– Intentionally put a root backdoor into bot– Losing points sucks
![Page 32: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/32.jpg)
Conclusions
• Live security exercises great for learning and security education
• They can be designed to create a research dataset
• Designed the 2010 iCTF to produce the first publically available dataset on CSA
• Presented SA metrics: toxicity and effectiveness
![Page 33: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/33.jpg)
Doupé - 7/12/11
Questions?
Data: http://ictf.cs.ucsb.edu/data/ictf2010/
Email: [email protected]: @adamdoupe
![Page 34: Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness](https://reader038.vdocuments.us/reader038/viewer/2022103115/55762650d8b42a4e1c8b50d4/html5/thumbnails/34.jpg)
Service Exploitation