jeremi gosney password cracking hpc passwords12
TRANSCRIPT
![Page 1: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/1.jpg)
Jeremi M Gosney Founder & CEO, Stricture Consulting Group Passwords^12 Security Conference December 3, 2012
![Page 2: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/2.jpg)
Password crackers need more power! • Yes, really.
A GPU is great!
More GPUs are better.
How many GPUs are enough?
The Problem
![Page 3: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/3.jpg)
Build up • $$ • Motherboard, BIOS, Driver limitations
Build out • Distributing load can be tricky
Potential Solutions
![Page 4: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/4.jpg)
Makes remote GPUs appear as if they were local
Implements entire OpenCL 1.1 Standard
Created by Amnon Barak and Amnon Shiloh, Hebrew University
Distributed by MOSIX (www.mosix.org)
Enter Virtual OpenCL (VCL)
![Page 5: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/5.jpg)
Free (gratis)
Supports any and all OpenCL devices
Works with any unmodified* OpenCL app
Eliminates the complexity of distributing load
Makes clustering ridiculously easy
VCL – Pros
![Page 6: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/6.jpg)
Closed source
Closed license
64-bit Linux only • (Not sure this is really a con) Need highspeed LAN – No Internet clustering
VCL – Cons
![Page 7: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/7.jpg)
Provides a pool of shared devices
Completely transparent to app & user
Two-Layer model • Broker node •Executes kernels on compute nodes.
• Compute nodes •Compute.
How VCL Works
![Page 8: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/8.jpg)
This is the front-end
• Has your software stack installed
• Only needs the VCL library and broker daemon
• Does not need any OpenCL devices
• Does not need any drivers
The Broker Node
![Page 9: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/9.jpg)
This is the back-end
• No software to install outside of VCL back-end daemon
• Needs OpenCL devices
• Needs proprietary drivers & OpenCL runtime
Compute Nodes
![Page 10: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/10.jpg)
VCL Architecture
Broker node
Back-end daemon
VCL Library
Broker
Compute node
OpenCL Devices
Back-end daemon
CPU Process
Kernels
Diagrams Courtesy of Amnon Barak
![Page 11: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/11.jpg)
Alternate Architecture
Compute node
OpenCL devices
Back-end daemon
VCL Library
Broker
Compute node
OpenCL Devices
Back-end daemon
CPU Process
Kernels
Diagrams Courtesy of Amnon Barak
![Page 12: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/12.jpg)
VCL Workflow
Broker node Compute node
Application
Kernel 1 running
Kernel 2 running
Kernel N running
File
System
Diagrams Courtesy of Amnon Barak
![Page 13: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/13.jpg)
Network optimization library for VCL
Single call for multiple executions
Direct file I/O to/from OpenCL memory object
Async data transfer with broker
SuperCL
![Page 14: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/14.jpg)
SuperCL Workflow
Broker node Compute node
Application Input data from file-system
Kernel 2 running Kernel 1 running
Kernel N running
Output data to file-system
File
System
Diagrams Courtesy of Amnon Barak
![Page 15: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/15.jpg)
We can use VCL for password cracking! • You knew this part was coming.
Jens Steube added VCL support for up to 128 AMD GPUs in oclHashcat-plus v0.09
MOSIX were more than happy to help debug and resolve issues
VCL + Hashcat
![Page 16: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/16.jpg)
Bandwidth
• Brute force / mask attacks need very little bandwidth
• Wordlist attacks need a lot
Latency
• Slow hashes can tolerate some latency
• Fast hashes cannot
• No SuperCL support in Hashcat
Memory
• Broker node needs a lot of it
• Higher the –n value, the more you need
VCL + Hashcat – Considerations
![Page 17: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/17.jpg)
Five 4U servers
25 AMD Radeon GPUs
• 10x HD 7970
• 4x HD 5970 (dual GPU)
• 3x HD 6990 (dual GPU)
• 1x HD 5870
4x SDR Infiniband interconnect
7kW of electricity
Broker daemon runs on a cluster node
Our Cluster
![Page 18: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/18.jpg)
epixoip@token:~/oclHashcat-lite-0.11$ LD_LIBRARY_PATH=/usr/lib/vcl vclrun \ ./vclHashcat-lite64.bin -b --benchmark-mode 1 –force oclHashcat-lite v0.11 by atom starting... Password lengths: 1 - 54 Watchdog: Temperature abort trigger disabled Watchdog: Temperature retain trigger disabled Device #1: Tahiti, 2048MB, 1100Mhz, 32MCU Device #2: Tahiti, 2048MB, 1100Mhz, 32MCU Device #3: Cypress, 512MB, 830Mhz, 20MCU Device #4: Tahiti, 2048MB, 1100Mhz, 32MCU Device #5: Cypress, 512MB, 830Mhz, 20MCU Device #6: Tahiti, 2048MB, 1100Mhz, 32MCU Device #7: Cypress, 512MB, 830Mhz, 20MCU Device #8: Tahiti, 2048MB, 1100Mhz, 32MCU Device #9: Cypress, 512MB, 830Mhz, 20MCU Device #10: Cayman, 1024MB, 880Mhz, 24MCU Device #11: Tahiti, 2048MB, 1100Mhz, 32MCU Device #12: Cypress, 512MB, 830Mhz, 20MCU Device #13: Cayman, 1024MB, 880Mhz, 24MCU Device #14: Tahiti, 2048MB, 1100Mhz, 32MCU Device #15: Cypress, 512MB, 830Mhz, 20MCU Device #16: Cayman, 1024MB, 880Mhz, 24MCU Device #17: Tahiti, 2048MB, 1100Mhz, 32MCU Device #18: Cypress, 512MB, 830Mhz, 20MCU Device #19: Cayman, 1024MB, 880Mhz, 24MCU Device #20: Tahiti, 2048MB, 1100Mhz, 32MCU Device #21: Cypress, 512MB, 830Mhz, 20MCU Device #22: Cayman, 1024MB, 880Mhz, 24MCU Device #23: Tahiti, 2048MB, 1100Mhz, 32MCU Device #24: Cypress, 512MB, 830Mhz, 20MCU Device #25: Cayman, 1024MB, 880Mhz, 24MCU [s]tatus [p]ause [r]esume [q]uit =>
![Page 19: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/19.jpg)
![Page 20: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/20.jpg)
![Page 21: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/21.jpg)
![Page 22: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/22.jpg)
![Page 23: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/23.jpg)
![Page 24: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/24.jpg)
![Page 25: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/25.jpg)
![Page 26: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/26.jpg)
Brute force consistently uses < 8 Mbps
Wordlist attacks on fast hashes use no more than 800 Mbps
Average peak of 88 Mbit per physical card
Ethernet latencies are still an issue • Infiniband helps tremendously
Bandwidth Measurements
![Page 27: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/27.jpg)
Benchmarks – Fast Hashes
0 50,000,000,000 100,000,000,000 150,000,000,000 200,000,000,000 250,000,000,000 300,000,000,000 350,000,000,000
NTLM
MD5
SHA1
LM
348 G/s
180 G/s
63 G/s
20 G/s
![Page 28: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/28.jpg)
Benchmarks – Slow Hashes
0 10000000 20000000 30000000 40000000 50000000 60000000 70000000 80000000
sha512crypt
bcrypt (05)
md5crypt77 M/s
71 k/s
364 k/s
![Page 29: Jeremi Gosney Password Cracking HPC Passwords12](https://reader034.vdocuments.us/reader034/viewer/2022050804/554091cb550346860b8b4ac8/html5/thumbnails/29.jpg)
IRC: epixoip on EFnet, Freenode
Twitter: @jmgosney
Keeping in touch