who am i? past: commercial waf developer since 2007 modsecurity maintainer 2007 – 2010 ids/ips...

Lockdown 2013

IronBee: Open source WAF engine with a commercial offering

Brian RectanusDirector of Engineering,

WAF

Lockdown 20132

Who am I?

Past:• Commercial WAF developer since 2007• ModSecurity maintainer 2007 – 2010• IDS/IPS Developer (OISF Suricata)

Present:• Lead WAF development @ Qualys in Madison• IronBee architect and developer

Lockdown 20133

What am I covering…

• Briefly: The what and why of WAF• IronBee, modules and rules• Overview of Qualys commercial WAF (beta)– How we use IronBee– How we have simplified the process– Beta features

Lockdown 20134

WAF: What is it?

• Web Application Firewall• To many this means:

“Block web based attacks.”

• But, WAFs are known to be a pain• There must be more

Lockdown 20135

WAF: Why does it exist?

• View inside your web applications• Log (and potentially block) suspicious activity• Block known and obvious attacks and tools• Limit attack surface• Buy time to fix problems• Feed your developers with more details• Deal with legacy products

Lockdown 20136

WAF: What gets in the way?

• Different interpretations of HTTP• Document types (HTML, XML, JSON)• Encodings (URL, Base64, entities)• Different Vectors (server, browser, DB, DoS)• Evasion techniques• Application logic (auth, sessions, BI)• Encryption, compression, obfuscation

Lockdown 20137

WAF: What can go wrong?

• False Positives (oops)• False Negatives (didn't see it)• Performance cannot suffer (too much)• Device failure (site is down)

Lockdown 20138

WAF: How can we make it better?

• Easier to setup and manage– Separate server/security configs and management– Low False Positives and low tuning costs– Flexible deployments with automated updates– Manage it all centrally

• Extensible engine– Solid framework for writing security logic– Integrate with other products– Combine many advanced techniques with correlation

• Acceptable performance– Intelligent application of security logic with fast algorithms

Lockdown 20139

IronBee: What is it?

• Open Source (Apache Software License v2)

github.com/ironbee

• Framework to inspect, block, modify and log• Extremely flexible• Highly extensible• Tries not to get in your way

Lockdown 201310

IronBee: Who is involved?Christopher Alfeld, PhD Mathematics and UW alumni

Experimental projects, performance, algorithms, C++ API

Sam BaskingerData structures, Configuration, Lua API

Nick Kew, Apache FoundationServer plugins: Apache Trafficserver, Apache httpd, nginx, tserver, …

Nick LeRoyCore engine, Testing

Brian RectanusInitial IronBee author, now architect and manager

Ivan RistićSecurity Research (SSL Labs – ssllabs.com, LibHTP, ModSecurity)

Many other supporting players at Qualys – too many to name here.

Lockdown 201311

IronBee: What's the basic concept?

• Server provides HTTP data– Web server, proxy, IDS, …

• Parsers break data into fields/streams– Headers, URI, POST body, cookies, …

• Modules/Rules inspect these fields/streams– Sigs, scoring, tracking, learning, correlation, …

• Actions performed:– Log, block, modify, track, …

Lockdown 201312

IronBee: What's a server?

• Provide HTTP data to IronBee• Implement blocking, modification (if possible)• Current:– Apache Trafficserver plugin– Apache Webserver module– Nginx plugin– Tserver (nginx fork) plugin– Clipp (command line with PCAP support)

Lockdown 201313

IronBee: What's the engine do?

• Notification of events• Core HTTP fields to inspect• Rule execution• Configuration• Logging

Very minimalistic, and becoming more so.

Lockdown 201314

IronBee: What are modules?

• Dynamically loadable shared libraries in C, C++• Minimal modules in Lua, but reloadable with config• Hook into IronBee events• Extend functionality (C/C++ only), such as:

– Parsers, normalizers, operators and actions– Rule languages (and extensions)– Embed scripting languages (Lua)– Enable technologies (libinjection - SQLi detection library)– Correlation (combine sigs, scoring, tracking, learning, …)– Logging– …

Lockdown 201315

IronBee: What are rules?

• Inspect data and perform actions• Simple signature language• Complex DSL (Lua @ config time)• Full scripting language (Lua @ runtime)• Extendible via modules

Lockdown 201316

Module: Simple Rule Language

Specify fields, inspect and perform an action:

Rule <fields> <op> <meta/actions>Rule REQUEST_HEADERS \

@rx "attack|pattern" \id:ex/1 rev:1 \phase:REQUEST_HEADER \event

Lockdown 201317


Transformations and meta data:

Rule REQUEST_HEADERS.count() \@gt 15 \id:ex/2 rev:1 \phase:REQUEST_HEADER \severity:75 confidence:80 \tag:http/limits \event

Lockdown 201318


Capture potential CC#s, blocking more than 10:

StreamInspect RESPONSE_BODY_STREAM \@dfa "\d{15,16}" \id:ex/3 rev:1 \capture:CC

Rule CC.count() \@gt 10 \id:ex/4 rev:1 \phase:RESPONSE_BODY \event block:immediate

Lockdown 201319


• These are just signature rules• Simple and come with limitations– Config file syntax (single line)– Somewhat verbose (requires id/phase)– No real flow control other than phase/file order

• Other types of rules eliminate these limits

Lockdown 201320

Module: Lua

• Embedded scripting language• As a configuration DSL (config time)• As a basic module (core engine runtime)• As a rule (rule engine runtime)

Lockdown 201321

Lua: As a DSL

DSL is named "waggle" (we like Bee themes here)

Rule REQUEST_HEADERS \@rx "attack|pattern" \id:ex/1 rev:1 \phase:REQUEST_HEADER \event

Sig("ex/1w", 1):fields("REQUEST_HEADERS"):op("rx", "attack|pattern"):phase("REQUEST_HEADER"):action("event")

Lockdown 201322

Lua: Programmatic Rules Config

Lua @ config time means full support for functions, loops, etc.

-- Parameterized rule with id/regexlocal function RequestRegex(id, regex)

return Sig("test/lua/" .. id, 1): fields("REQUEST_HEADERS”): op("rx", regex): phase("REQUEST"): actions("event”)end

-- Simplify management and readabilityRequestRegex(1, [[attack|pattern]])RequestRegex(2, [[attack2|pattern2]])

Lockdown 201323

Lua: Basic Modules

Lua executed at runtime to handle core engine events.

-- Get the IronBee Module object.local ibmod = ...

-- Define a function to handle an event.local function log_event(ib) ib:logInfo("Handling event=%s”, ib.event_name) return 0end

-- Register to be called with the event.ibmod:request_header_finished_event(log_event)

Lockdown 201324

Lua: Rules

• Similar to Lua module, but less complex• Lua executed by the rule execution engine• Entire script runs vs. using event callbacks

Lockdown 201325

Rules: Scaling to the non-trivial

• Simple linear execution with basic rules– Executes a list of rules per phase– All rules are executed

• What about 1000s or 100,000s of rules?• Need a way to limit execution• Need a way to specify dependencies/order• Need a way to cache results• Need a higher level of logic and correlation

Lockdown 201326

Rules: Made to be extended

• Rule injection• Modules can take ownership of rules• Modules can decide if/when rules execute• Currently two modules use this facility– Fast rules module– Predicate rules module

Lockdown 201327

Module: Fast Rules

• Adds a fast pattern (prequalification) to rules• Rules are executed only if prequalified• All fast rules utilize modified Aho-Corasick– Extensions to utilize fixed width patterns– Speed is independent of number of patterns– Works best with large rulesets– Some limitations

Lockdown 201328

Fast Rules: An example

Utility suggests fast patterns for existing rules by adding comments to rules

# FAST RE: ^(.+),\s*max-age[^,]+,?(.*)$# FAST Suggest: "fast:max-age[^,]"Rule RESPONSE_HEADERS:Cache-Control \ @rx "^(.+),\s*max-age[^,]+,?(.*)$" …

Rule RESPONSE_HEADERS:Cache-Control \ @rx "^(.+),\s*max-age[^,]+,?(.*)$" "fast:max-age[^,]" …

Lockdown 201329

Module: Predicate Rules

• Uses Lua DSL to produce predicate expressions(and (gt (atoi (field 'Content-Length')) 0) (streq 'GET' (field 'Request-Method')) )

• Complex rules are built from simple rules• Rules form an knowledge graph– Graph optimizations performed at configuration time– Common sub-expression merging & caching– Only required rules execute, and only once

• Combines Lua DSL and runtime optimizations– Full Lua support enhances configuration– Graph optimizations enhance runtime

Lockdown 201330

Predicate Rules: Named predicates-- Parameterized named predicatelocal function header(name)

return P.Field('REQUEST_HEADERS'):sub(name)end-- Named predicateslocal range_header_too_long = P.Gt(header('Range'):length(), 1000)local host_header_too_long = P.Gt(header('Host'):length(), 100)

-- Combine named predicates into a rule/signature-- NOTE: A "/" operator is overloaded for predicates to P.Or(…)Sig(”ex/p/1", 1):

predicate( range_header_too_long / host_header_too_long ):phase([[REQUEST_HEADER]]):action([[event]]):message([[Invalid HTTP header: too long.]])

Lockdown 201331

Predicate Rules: Lua DSL in actionlocal sensitive_file_patterns = { unix = [[(?:/etc/passwd|/etc/hosts|/etc/shadow|/bin/id)$]], java = [[(?:WEB-INF/web.xml|/conf/server.xml)$]], apache = [[(?:.htaccess|.htpasswd|.meta|.web)$]]}local function contains_sensitive_files(pattern) local r = P.false for i,v in ipairs({"REQUEST_URI_PATH", "REQUEST_HEADERS", "ARGS"}) r = P.Or(r, P.rx(pattern, P.Field(v):remove_whitespace())) end return rendfor name,pattern in pairs(sensitive_file_patterns) do Sig("qrs/LFi/" .. name, "1"):

predicate(contains_sensitive_files(pattern)):phase([[REQUEST_HEADER]]):action([[event]]):message("LFi: request for sensitive " .. name .. " files.")

end

Lockdown 201332

Framework: Automata

• Iron Automata (we also like Iron themes here)• Framework and utils for building automata• Splits generation, optimization, execution• Generic execution environment, Eudoxus• Example Automata: Enhanced Aho-Corasick– Caseless matches– Fixed width patterns/sets (char sets, negation– Can be tuned for space vs time through Eudoxus

Lockdown 201333

IronAutomata: Aho-Corasick Example1

• Aho-Corasick• Unoptimized• Patterns:– he– she– his– hers

Lockdown 201334

IronAutomata: Aho-Corasick Example2

• Aho-Corasick• Speed

Optimized• Patterns:– he– she– his– hers

Lockdown 201335

IronAutomata: Optimization

• Aho-Corasick• Patterns:

~250k English Dictionary

• Data:Text of "Pride and Predjudice" novel 10x

Lockdown 201336

Module: Eudoxus Executor

Execute compiled, eudoxus automata.• Large signature database– Spam keywords– Known attack patterns– Link reputation

• Custom, auto generated automata– Based on research– Based on website traffic profiling

Lockdown 201337

Utility: Clipp

• Command line utility• Testing and rule development• HTTP data via: Raw files, PCAP, protobuf, …• Modify HTTP data via filters• Convert between formats• Highly extendible• Ruby wrapper for unit/regression testing

Lockdown 201338

IronBee: Batteries not included

• Management is not dictated, so…• No Config Management• No Rule Management• No Log Management• Must do these yourself– You should already be doing this– The point is to stay out of your way– Allow you to use your own management tools

Lockdown 201339

Qualys WAF: What will it add?

• Managed WAF appliances via cloud• Automated updates– Software– Modules– Rules

• Integration with other Qualys products– Web Application Scanning– Asset Management

Lockdown 201340

Qualys WAF Beta: What's offered?

• Initially Amazon Web Services Platform– EC2 Classic and VPC– Clustering via ELB– Auto-scaling– You decide how much power you need

• We are expanding to other platforms

Lockdown 201341

Qualys Beta WAF: What's it do?

• Manage AWS based WAF Appliances• Generic attack detection• Declarative security (fixup cookies/headers)• Data leakage detection• Reduce attack surface (HTTP limitations)• ACLs (IP and geo)

Lockdown 201342

Qualys WAF Beta: What's it look like?

• Manage AWS Appliances• Manage events• Generic attack detection• Declarative security• Data leakage detection• Reduce attack surface• Access Control

Lockdown 201343

Qualys WAF Beta: AppSec

Lockdown 201344

Qualys WAF Beta: InfoLeak

Lockdown 201345

Qualys WAF Beta: Fixups

Lockdown 201346

Qualys WAF Beta: HTTP

Lockdown 201347

Qualys WAF Beta: ACLs

Lockdown 201348

Qualys WAF: What's coming?

• QualysGuard integration– WAS scan result feedback– Shared assets

• False positive mitigation• Exception handling• Website and session profiling• Reporting

Lockdown 201349

We are Hiring in the Madison!

• Product Management• Application Security Researchers• Developers• QA

Contact me if you are interested.

Lockdown 201350

Thanks!

github.com/ironbeequalys.com/waf

qualys.com/careers

Feel free to contact me for more info.

Brian [email protected]

who am i? past: commercial waf developer since 2007 modsecurity maintainer 2007 – 2010 ids/ips...

Documents