Master Serial Killer

Chris Sistrunk PE, Mandiant

Adam Crain, Automatak

About Us

Chris Sistrunk, PE• Electrical Engineer• SCADA Expert• Loves Security• DNP3 Member• Button Pusher

Adam Crain• Software Engineer• OSS Advocate• openDNP3 Author• DNP3 Member• Code Monkey

How I Audit SCADA systems

http://securityreactions.tumblr.com/post/30866100673/how-i-audit-scada-systems

ICS/SCADA Security

• ICS/SCADA lags IT by 10-15 years• 708 SCADA-related vulns on OSVDB.org

since 2011. “Like kicking a puppy”• Positive vs. Negative Testing: The front

yard is mowed, but the back yard is overgrown.

Software Testing

When you scan ICS with nmap

SCADA Protocol Vuln Research

We chose to focus on popular SCADA protocols

Fuzzers did exist, but only tested server side

Serial had not been fuzzed before (that we know of)

We chose to use Responsible Disclosure• Inform the vendor, then ICS-CERT, DNP3 UG• Worked with the vendor to help them replicate and

begin further negative testing

Project Robus• Latin for “bulwark”• Started in April 2013• 24 advisories / 30 tickets• 22 DNP3, 1 Modbus,

1 Telegyr 8979

www.automatak.com/robuswww.automatak.com/aegis

Fuzzing Master Stations

• Referenced in Nat’l SCADA Test Bed reports but no data available

• Wurldtech & Spirent (Mu Dynamics) don’t fuzz the master side of ICS protocols…………..yet

Master Slave

Fuzzing Master Stations

DNP3 Application Function Code 0x82• If the Master Station has Unsol enabled, it must accept

messages from its RTUs at any time• Design of System must be fine tuned...or else

DNP3 Outstation Unsolicited Response Storm• If the Master parser has problem with one message,

you can imagine the problems with many many messages

Serial Fuzzing

All the security focus has been on ethernet networks, but many ICS, especially SCADA, still utilize serial networks.

• DNP3 is same! (unlike Modbus)• Impact to NERC/CIP v3 & v5

Physical Security (discuss later)• Pole-mounted RTUs• PQ Meters, etc

DNP3 (IEEE 1815-2012) Primer

DNP3 is a SCADA protocol used by almost all of the electric utilities and some water in North America, Australia, and the UK.

Created in 1990s and turned over to DNP3 UG in 1993.

One of the few ICS protocols that has secure auth.SCADA Master

RTU with I/O

Breaking Down DNP3

Ref from IEEE Std 1815-2012

TCP 20000TCP 19999 (TLS)UDP 20000

Courtesy of

Vendor Response MatrixICS-CERT Adv Company Protocol Bug Fix Days AdvisoryICSA-13-161-01 IOServer DNP3 4/24 5/24 30 6/10/2013ICSA-13-213-03 IOServer DNP3 5/1 7/20 80 8/1/2013ICSA-13-219-01 SEL DNP3 5/1 5/30 29 8/7/2013ICSA-13-226-01 Kepware DNP3 4/24 6/18 55 8/14/2013ICSA-13-234-02 TOP Server DNP3 4/24 6/18 55 8/22/2013ICSA-13-240-01 TMW DNP3 4/24 6/17 54 8/28/2013ICSA-13-213-04A Matrikon DNP3 4/24 6/17 54 8/29/2013ICSA-13-252-01 Subnet DNP3 4/24 8/30 128 9/9/2013ICSA-13-282-01 Alstom DNP3 4/24 6/4 41 10/21/2013ICSA-13-297-01 Catapult DNP3 4/24 10/1 160 11/22/2013ICSA-13-297-02 GE IP DNP3 Self Report 10/1 n/a 11/22/2013ICSA-13-337-01 Elecsys DNP3 9/12 11/4 53 12/3/2013ICSA-13-346-02 Cooper OPC DNP3 7/31 None ∞day™ 12/12/2013ICSA-13-346-01 Cooper/Cybectec DNP3 5/1 12/12 225 12/12/2013ICSA-13-352-01 Novatech DNP3 5/1 9/5 127 12/18/2013ICSA-14-014-01 Schneider DNP3 8/6 8/23 17 1/14/2014ICSA-14-100-01 IOServer Modbus 2/6 3/4 26 4/10/2014ICSA-14-154-01 COPA-DATA DNP3 Self Report using Aegis! n/a 6/3/2014ICSA-14-196-01 Subnet TG8979 4/18 6/18 61 7/31/2014

Vendor Response

• Most of the vendors were very pleased• A few were not >> head in the sand• Some had never done negative testing• Nearly all devices and hosts with DNP3

were affected, so it was an industry-wide wakeup call.

White Noise Fuzzing

#1 random == really “dumb”

Template (mutational) Fuzzing

Generational “Smart” Fuzzing

Multi-field Anomalies

Hanna Jack I’m

!everybody

Hi

Generational == most vulns!

The Aegis ICS Fuzzing Framework

• We decided that we needed to release our fuzzing framework tool as open source.

• Open source security tools have a proven track record of raising security (hello MSF!)

• We do encourage people to join our efforts to add more protocols to Aegis

Aegis Specifics

• Version 0.1.x in Scala www.scala-lang.org• Current version (private release) in C#• Protocol boundary conditions• Abstracts physical layer• Combines aspects of generation and mutation• Repeatable random seeds• ~500,000 test cases with one seed

Test DNP3 Message (DL, TL, or AL)

Request Link States

Link Status

x Num Test Cases

Request

Response

x Num Retry (10)

Fuzzer Test Flow

I 0x0564 U...

Y U NO 0x0564 ME BAK ?!

val nums = List(1, 3)

val colors = List(“red”,”green”)

// repeat the reversed string num times

def combine(i: Int, s: String) = List.fill(i)(s.reverse).mkString

val result = Cartesian.Transform(colors,nums)(combine)

What is result?

Combinatorics

// val nums = List(1, 3)

// val colors = List(“red”,”green”)

> result.foreach(println)

der

derderder

neerg

neergneergneerg

Lazy Generator

{ frames } = f (byte,Type)

{ 0, 1, 63 }{ true, false } { true, false }

{byte} = f (bool, bool, int) {Type} = f (.....)

...........................

Fuzzing is O(2n)

Generators can get large!

{ test cases } ● Many function codes

● Many objects

● Header types

● Many field values

Types of Vulnerabilities

FA 82 00 00 01 00 02 00 00 00 00 FF FF FF FF

UnsolicitedResponse

Group 1Variation 0

Sizeless?!

4 byte start/stop

● infinite loop● missing data● integer overflow?● accepts broadcast

0 4294967295

Vuln #1

DD 82 00 00 0A 02 01 00 00 FF FF

UNSOL

Group 10Variation 2

Binary OutputStatus

2 byte start/stop

● infinite loop● missing data● unexpected data● integer overflow?

0 65535

Vuln #2

05 64 06 44 64 00 64 00 FF F2 C0 1D 0A

1 byte payload

● transport header only● unhandled exception

100 100

unconfirmeduser data

CRC CRC

FIR / FINSEQ = 0

Vuln #3

Recorded Demos

Video 1: a DNP3 outstation

-application layer object fuzzing

Video 2: a DNP3 master

-unsolicited application layer fuzzing

DD 82 00 00 0C 01 00 00 01 rnd(11) rnd(11)

UnsolicitedResponse

Control RelayOutput Block

1 byte start/stop

● buffer overrun● not malformed!● unexpected objects● accepts broadcast

CROB #1 CROB #2

Vuln #4 (TMW integration)

FA 82 00 00 02 02 01 01 00 FF FF

UnsolicitedResponse

Group 2Var 2(event)

2 byte start/stop

● stable infinite loop● max range - 1 and no data● accepts broadcast

1 65535

Vuln #5 (TMW integration)

Using Aegis

So easy…Twitter can do it

ExamplesRun 10 link layer test cases starting at #123

$ aegis-console -mid dnp3 -pid lfuzz -start 123 -count 10

Unsolicited response fuzzing of a master listening on default port 20000 with master address of 0 and an outstation address of 1

$ aegis-console -mid dnp3 -pid aufuzz -dest 0 -src 1 -master -listen

Outstation link layer fuzzing test case #100 only

$ aegis-console -mid dnp3 -pid lfuzz -start 100 -count 1

Outstation application object fuzzing against 192.168.1.55:20001 with default addressing

$ aegis-console -mid dnp3 -id aofuzz -host 192.168.1.55 -port 20001

Further Aegis Development

• In addition to DNP3 protocol, we’ve added Modbus and Telegyr 8979 (serial only) protocol modules to the framework.

• Migrated from scala to C#.• Added a GUI• Working with vendors and other trusted

researchers.

New Aegis Demo --- module: dnp3 - Test routines for the DNP3 protocol ---

Procedure ids:

link Fuzzing of the link layer (masters or outstations)transport Fuzzing of the transport function (masters or outstations)requests Fuzzes the application layer with malformed and unexpected requests (outstation)unsol Fuzzes the application layer with malformed and unexpected unsolicited responses (master)octetunsol Reports large numbers of 0-length octet string headers via unsolicited mode (master)octetwrite Writes large numbers of 0-length octet string headers (outstation)randrequest Fuzzes the application layer with semi-random requests (outstation)randunsol Fuzzes the application layer with semi-random unsolicited responses (master)

-dest <arg>(1024)[0, 65535] link layer address of the target-src <arg>(1)[0, 65535] link layer address of the fuzzer-master <arg>(False) set the link-layer master bit for master fuzzing-retries <arg>(10)[1, none] Number of link status retries-timeout <arg>(1000)[10, none] Read timeout in milliseconds-health <arg>(LinkStatus) Type of health check to use [linkstatus, resetlink]

--- module: modbus - Test routines for the Modbus protocol ---

Procedure ids:

request Sends malformed or unexpected requests at a Modbus slave

ICS/SCADA Defense

ICS/SCADA Defense

Network Security Monitoring (do it now!)• Bro, SNORT, Wireshark can parse DNP3 & Modbus!• Deep packet inspection firewalls• Full packet capture (even serial) – 1TB y’all• Use Security Onion to monitor ICS networkshttp://www.liquidmatrix.org/blog/2014/07/01/is-there-a-cuckoo-in-your-control-system/

tl;dr ≥1 person, Security Onion, and an ICS Honeypot

ICS/SCADA Defense

• Install patches – not quite like IT• Robust device & network configuration

• Disable unused ports, protocol function codes• Whitelist apps and even traffic

• DNP3 Secure Authentication v5 & TLS• Signed software/firmware

Physical Security

3/8” Mesh

ASTM Grade 6Buys extra time

What’s different about Robus?

SCADA Vulns reported for a while now

Adam and I aren’t security researchers• He’s a software geek…I’m an engineer• Our skills complemented each other• Both experts in DNP3 protocol, but from

different angles

Some theories

Why did the industry move instead of ignore?• I was an end user and we really cared!• Not just a wham-bam researcher• Respectful, tactful, responsible• We released our tool

…………we weren’t going away

I’m still more worried about…

SHODANProbably default configs

• Many similar responses

• Same DNP Addressespython shell

>>> " ".join("%02x" % ord(i) for i in "DNP3 paste from shodan”)

Unsolicited Response with Binary and Analog Data

Class 1/2/3/0 Poll!!!

https://ics-radar.shodan.io/

https://maps.shodan.io/

Conclusions

• DNP3 is not a special case, other protocols same fate

Modbus, IEC 60870, IEC 61850, ICCP, EtherNet/IP…• Early testing both slave/server AND master/client sides

of protocols are important!• Compliance != Security, but the culture is important• Don’t have to be a nation/state or large firm to do this• A few good folks can make a difference in the industry

Questions?

@jadamcrain@chrissistrunk