network attack injection - di.fc.ul.ptnuno/thesis/joaoantunes_phd_maio12.pdf · analyzing the...

UNIVERSIDADE DE LISBOA

FACULDADE DE CINCIAS

DEPARTAMENTO DE INFORMTICA

NetworkAttackInjection

JooAlexandreSimesAntunes

DOUTORAMENTOEM INFORMTICA

ESPECIALIDADE CINCIA DA COMPUTAO

2012

http://www.ul.pthttp://www.fc.ul.pthttp://www.di.fc.ul.ptmailto:[email protected]

UNIVERSIDADE DE LISBOA

FACULDADE DE CINCIAS

DEPARTAMENTO DE INFORMTICA

NetworkAttackInjection

JooAlexandreSimesAntunes

TeseorientadapeloProf. DoutorNunoFuentecillaMaiaFerreiraNeves

especialmenteelaboradaparaaobtenodograude

doutoremInformtica, especialidadedeCinciadaComputao.

2012

http://www.ul.pthttp://www.fc.ul.pthttp://www.di.fc.ul.ptmailto:[email protected]

Abstract

Theincreasingrelianceonnetworkedcomputersystemsdemandsfor

high levelsofdependability. Unfortunately, new threatsand forms

ofattackareconstantlyemergingtoexploitvulnerabilitiesinsystems,

compromisingtheircorrectness. Anintrusioninanetworkservermay

affectitsusersandhaveseriousrepercussionsinotherservices, possi-

blyleadingtonewsecuritybreachesthatcanbeexploitedbyfurther

attacks. Softwaretestingisthefirstlineofdefenseinopposingattacks

becauseitcansupportthediscoveryandremovalofweaknessesin

thesystems. However, searchingforflawsisadifficultanderror-prone

task, whichhasinvariablyoverlookedvulnerabilities.

Thethesisproposesanovelmethodologyforvulnerabilitydiscovery

thatsystematicallygeneratesandinjectsattacks, whilemonitoringand

analyzing the target system. Anattack that triggersanunexpected

behaviorprovidesastrongindicationofthepresenceofaflaw. This

attackcanthenbegiventothedevelopersasatestcasetoreproduce

theanomalyandtoassistinthecorrectionoftheproblem.

Themain focusof the investigation is toprovidea theoretical and

experimentalframeworkfortheimplementationandexecutionofat-

tackinjectiononnetworkservers. Severalinnovativesolutionsrelated

tothisapproacharecovered, includingwaystoinferaspecification

oftheprotocolimplementedbytheserver, thegenerationofacom-

prehensivesetofattacks, theinjectionandmonitoringofthetarget

system, andtheautomaticanalysisofresults.

Furthermore, weapply someof thedeveloped techniques toother

areasofnetworksecurity, namelytointrusiontoleranceanddetection.

Inparticular, anewmethodisproposedtoassistontheevaluationof

thecomplianceofdiversereplicasinintrusion-tolerantsystems.

Keywords: Network Security, Network Servers, Vulnerabilities, At-

tackInjection, Monitoring, ProtocolReverseEngineering, TestCase

Generation, BehavioralInference

Resumo

O aumentodadependnciaeconfianadepositadanossistemasde

rede, exigenveisdeconfiabilidadecadavezmaiselevados. Infe-

lizmente, novasameaaseformasdeataqueestoconstantementea

surgir, explorandovulnerabilidadesnossistemasecomprometendo

asuacorretaoperao. Uma intrusonumservidorde redepode

afetarosseusutilizadoresetergravesrepercussesnoutrosservios,

eventualmenteabrindonovasbrechasdeseguranaquepodemser

exploradasporoutrosataques. O testedesoftwareaprimeiralinha

dedefesanaoposioaataquesporquepodeapoiaradescobertae

remoodefraquezasdossistemas. Noentanto, aprocuradefalhas

uma tarefadifcil epropensaaerros, eque tem invariavelmente

deixadoescaparvulnerabilidades.

A tesepropeumanovametodologiaparaadescobertadavulnera-

bilidadesquepermiteasistemticageraoeinjeodeataques, ea

simultneamonitorizaoeanlisedosistema-alvo. Umataqueque

desencadeieumcomportamentoinesperadoumaforteindicaoda

presenadeumafalha. Esteataquepodeentoserdadoequipade

desenvolvimentocomoumcasodetesteparareproduziraanomalia

eparaauxiliarnacorreodoproblema.

O focoprincipaldainvestigaofornecerumquadrotericoeex-

perimentalparaaconcretizaoeexecuodainjeodeataques

emservidoresde rede. Diversas solues inovadoras relacionadas

com esta abordagem so estudadas, incluindo a inferncia da es-

pecificaodoprotocoloconcretizadopelo servidor, ageraode

umconjuntoabrangentedeataques, a injeoemonitorizaodo

sistema-alvoeaanliseautomticadosresultados.

Almdisso, aplicamosalgumasdastcnicasaquidesenvolvidasnou-

trasreasdeseguranaderedes, nomeadamente, paraatolerncia

edeteodeintruses. Emparticular, propostoumnovomtodo

paraaavaliaodaconformidadederplicasemsistemastolerantes

aintrusescomdiversidade.

Palavras-Chave: SeguranaemRedes, ServidoresdeRede, Vulnerabi-

lidades, InjecodeAtaques, EngenhariadeReversodeProtocolos,

GeraodeCasosdeTeste, InfernciaComportamental

ResumoAlargado

Aolongodoltimosculoassistiu-seaumcrescentenmerodeinovaes

queforamdecisivasparaoavanodoconhecimentoedatecnologia. A Internet

foiumadessasinovaesqueteveumimpactoprofundonasociedadeerevolu-

cionouomodocomocomunicamosevivemos.

HojedependemosdaInternetedosseusvariadosserviosparamuitomais

quea simplesbuscaepartilhade informao. Estima-sequeem2011on-

merodeutilizadoresdaInternettenhaultrapassadoos2,4milmilhes, maisdo

dobrodoqueem2006(ITU, 2011). Nosasuapopularidadetemvindoaau-

mentar, comotambmumgrandeimpulsionadoreconmico, movimentando

quase6biliesdeEurosporanonomundointeiro. Calcula-semesmoquea

Internetrepresenteemmdiacercade3%doPIB dospasesdesenvolvidosou

emfortecrescimentoeconmico, ultrapassandomesmoossetoresdaagricultura

edaenergia(Rausaset al., 2011).

A Internetdestepontodevistaumservioessencialsociedadeeecono-

mia, criandoempregoeconhecimento, alterandoestilosdevidaeatafetandoa

polticainternacional, evidenciadonasrecentesquedasdevelhosregimes(Coz,

2011). EstadependnciaaindaagravadapelofactodaInternetservirdesuporte

aalgumasinfraestruturascrticas, semasquaisasociedadecomoaconhecemos

nopoderiasubsistir. Ostransportes, acomunicao, adistribuiodeenergia

eltricaeatividadesbancriasefinanceirassoalgunsdosserviosdosquaisde-

pendemosequesocontroladoseoperadosatravsdaInternet. A interrupo

deumdestesserviospodetergravesconsequncias(US-CanadaPowerSystem

OutageTaskForce, 2006; Castle, 2006), tornandoaInternetnumpontodede-

pendncianicoeextremamentesensvel, queraataquesfsicos, queraataques

informticos.

A exploraomaliciosadeumservidornaInternetpodeafetarnoapenas

osseusutilizadores, comotambmdespoletarumefeitoemcascatacomgraves

repercussesnoutrosservios(Sharma, 2010). Vriosestudoseinquritosreve-

laramqueonmerodeataquesinformticostemvindoaaumentar, sendocada

vezmaiseficientesesofisticados(DobbinsandMorales, 2011; McAfee, 2011;

Symantec, 2012). Estesataquesjnososimpleseinofensivasdemonstraes

depoderentregruposdepiratasinformticos, massoesforosbemconcerta-

dos, organizadosefinanciados. Assuasmotivaessovariadas, podendoirdo

lucropessoalarazespolticasoucomerciais, envolvendoporvezesasabota-

gemeespionagemindustrialougovernamental(Garget al., 2003; Andersonand

Nagaraja, 2009; Krekel, 2009; Kanichet al., 2011; Rashid, 2011).

Defenderosservidoresderededeataqueseevitarquesejamexploradosma-

liciosamentetorna-seassimessencialaocorretofuncionamentodaInternet. Po-

rm, astarefasrelacionadascomaseguranasomuitasvezesdelegadaspara

segundoplano, emparteporquesovistascomomenosfundamentaisquando

comparadascomainclusodenovasfunes. Defacto, amaiorpartedoesforo

nodesenvolvimentodesoftwarevaitipicamenteparaaintroduodenovasfun-

cionalidades, umavezqueestassocomercialmenteapelativas. Istotorna, por

exemplo, asoperaesdetestecadavezmaiscomplexasedispendiosas, visto

haverummaiornmeroderequisitosfuncionaisaserverificado, bemcomore-

quisitosno-funcionais, taiscomogarantirnveisaceitveisdedesempenhoede

confiabilidade.

O testedesoftwareaprimeiralinhadedefesanaprevenoeresistnciaa

ataquespoissuportaadescobertaeremoodevulnerabilidades. Todavia, pro-

curarporerrosemaplicaeseservidoresumprocessometiculosoemoroso,

sendotradicionalmenterealizadoporsereshumanos. Porconseguinte, osmto-

dosclssicosdetestetminvariavelmentedeixadoescaparerros. Poroutrolado,

osmtodoslevadosacabopelosatacantesnadeteoeexploraodevulne-

rabilidadestm-sereveladobastanteeficazes, oquebemvisvelnocrescente

nmerodevulnerabilidadesreportadas(Fossiet al., 2011).

Comoresultado, hanecessidadedesecriaremnovosmecanismosparaa

anlisedos sistemas, emparticularnoquediz respeitoa vulnerabilidadesde

segurana. Estetrabalhodedoutoramentovisacontribuircomumanovaabor-

dagemparaadescobertadevulnerabilidadesqueassentanaideiadequeuma

vulnerabilidadeexistentenosistemassemanifesta, atravsdeumerrooude

umafalha, quandoativadaporumataqueespecfico. A metodologiadeinjeo

deataquesaquipropostaprocurasolucionaroproblemadadescobertadevul-

nerabilidadesdeumaformaautomatizada, atravsdageraoeinjeodeum

conjuntodeataques(oucasosdeteste)comomximodecobertura. A automa-

tizaodoprocessodetestepermitepotencialmenteobterganhossignificativos,

nosnadiminuiodoscustos, queemalgunscasospodemchegara50%dos

custos totaisdedesenvolvimento (Beizer, 1990), mas tambmnaobtenode

resultadosmaiscompletoseconfiveis.

Noentanto, este tipode soluo traz consigo vrios desafios, requerendo

soluescomplexasqueabordamvriosdomniosdeinvestigao, desdeain-

duodegramticasmonitorizaodeaplicaes, passandopelageraoe

execuodecasosdeteste. A soluopropostanateseenglobaporissodiver-

sasfases, concretizadaspordiferentescomponentes, responsveisporsolucionar

necessidadesespecficasdametodologia.

Numaprimeirainstncia, pretende-seobterumaespecificaodosistema-

alvoquemodeleoseucomportamentoemrelaoaomodocomoforneceo

seuservio. Nocontextodainjeodeataquesderede, osistema-alvoum

servidorquetrocamensagensderedecomosseusclientesdemaneiraaoferecer

oservio. Portanto, amodelizaodoservidorpodeseraproximadaatravsda

especificaodoprotocolodecomunicaoqueconcretiza. A obtenodesta

especificaoentoumcomponentefundamentaldametodologia, umavez

queosataquesutilizamcomoveculodetransporteoenviodemensagensem

conformidadecomaqueleprotocolo.

Emalgunscasos, aespecificaodoprotocolopodenoestaracessvel, como

porexemploseoprotocoloforproprietrio(oufechado). Mesmoemprotoco-

losabertos, traduziradocumentaoquedescreveaespecificaonummodelo

formalumprocessorelativamentemorosoepropcioaerros. A teseprope

umasoluoparaaobtenodaespecificaodeumprotocolodeumaforma

automatizada. A soluoseguidaassentanuma tcnicadeengenhariade re-

versoqueutilizaapenasasmensagensdoprotocolotrocadasentreclientese

servidores. A abordagembaseia-senoproblemadeinduodegramticase

aplicadaparainferiralinguagemdoprotocolo(i.e., quaisasmensagensvlidas

eosseusrespetivosformatos)easuamquinadeestados(i.e., quaisasrelaes

estabelecidasentreosdiferentestiposdemensagens).

A especificaodoprotocolodosistema-alvodepoisutilizadapelameto-

dologianoprocessodegeraodeataques. Talcomonaperspetivadeumad-

versrio, oconjuntodeataquesdeverserexaustivo, demodoaobterumaboa

coberturadeteste, masnodemasiadonumerosoaopontodetornarasuainje-

oimpraticvel. Duasaproximaesparaageraodeataquessoestudadas

natese. A primeiradefinequatroalgoritmosdegeraodecasosdetestequepro-

curamexperimentardiferentesaspetosdaconcretizao, desdemensagenscom

errosdesintaxeamensagenscomcontedopotencialmentemalicioso. A se-

gundaabordagemprocurareaproveitarcasosdetesteexistentes(e.g., produzidos

porferramentasdesegurana)paragerarataquesespecficosparaosistema-alvo.

Oscasosdetesterecicladospodemattersidodefinidosparaoutrosprotocolos.

Logo, estasoluopodepermitirtestarsistemas-alvoqueconcretizemnovospro-

tocolos(ouextenses)equeporissoaindanososuportadospelasferramentas

existentes.

Umavezdefinidososcasosdetese, conduzidaumacampanhadeinjeo

deataquesnumambientecontroladoemqueoscasosdetestesoexecutadosno

sistema-alvo. A injeorealizadaapardamonitorizaodoservidor, demodo

adetetarqualquercomportamentoanmaloquepossaindiciaraativaodeuma

vulnerabilidade. A monitorizaoumaspetoessencialnainjeodeataques

epor isso foramdefinidasdiferentesmaneirasdeaconcretizar. Doisaspetos

essenciais tiveramde ser tomados emconsiderao. Por um lado, o tipode

monitorutilizadopodedeterminarasclassesdevulnerabilidadesquesepodem

detetar. Poroutrolado, quantomaioracapacidadedomonitor, maisrequisitos

soexigidosaoseuambientedeexecuo, oquepodetornarasuaoperao

demasiadocomplexae/ou lenta (i.e., podeafetarocorreto funcionamentodo

sistema-alvoearespetivaavaliao).

O passofinalna injeodeataquesprocurarpor indciosdeanomalias

nocomportamentodosistema-alvo. Vistoqueaexecuodosistema-alvodi-

tadapelaformacomoesteprocessaosataques, qualquercomportamentofora

doesperadoindicaqueumapotencialvulnerabilidadefoidespoletada. O ata-

quequecausouoerro(tipicamenteoltimo)poderentoserutilizadocomo

casodetestequepermitirreproduziraanomaliaelocalizaravulnerabilidade

(e.g., erronocdigoounaconfigurao). Doistiposdeabordagensforamaqui

seguidos. Noprimeirosodefinidasvriasclassesdecomportamentoanmalo,

tpicasdevulnerabilidadesconhecidas, quedepoissoprocuradasnasrespostas

enamonitorizaodosistema-alvo. Estasoluopermitedescobrirrapidaeau-

tomaticamentevriostiposdevulnerabilidadescujosefeitossobemconhecidos

(e.g. crash).

A segundasoluo, estendeaespecificaodoprotocolocomdadosdemo-

nitorizaointerna, demodoadefinirocomportamentocorretodosistema-alvo

para todoo espaodoprotocolo. Esta especificao estendida, chamadade

perfil comportamental, depois comparada coma execuodo sistema-alvo

enquantoprocessaosataques. Qualquercomportamentodesvianteentoas-

sinaladocomocausadoporumapotencialvulnerabilidade. Umaanliseexpe-

rimentalutilizandoestaabordagempermitiuconcluirqueadefiniodoperfil

comportamentalpermitedetetarosdiferentestiposdeexecuofaltosa.

Algumasdasideiasaquiapresentadasforamtambmaplicadasnacriaode

outrotiposdemecanismosdesegurana. Nomeadamente, assistindonacriao

deumsistematoleranteaintrusesutilizandodiversidade, enodesenvolvimento

desistemasdedeteodeintrusesmaiscapazeseadequadosasistemascrticos.

Acknowledgements

Firstandforemost, I wanttoacknowledgemyadvisor, ProfessorNuno

FerreiraNeves. Hissupportandguidancewascrucialtothesuccess

of thisdoctoralwork, andI amappreciativeof theconfidenceand

freedomhehasindulgedmethroughoutthisjourney. Hehasbeen

aninspirationforpursuingperfectionandforbecomingabetteraca-

demicresearcher. Thankyou!

I would also like to thankProfessor PauloVerssimo. He is a cor-

nerstoneintheresearchstatusofthegroupandhisstrongleadership

andinsightsareevidentthroughouttheaccomplishmentsofboththe

Navigatorsteamandofitscurrentandpastmembers.

I would like toacknowledge theUniversityofLisboa. Thesewalls

haveshapedandformedmanymenandwomensince1911andit

vowstoleaditsstudents adlucem (tothelight)ofknowledge. This

hasbeenmyhomeformorethanadecadeandisnotwithoutamisty

eyethatI nowleave. I wouldliketothankafewnotableindividu-

alsfromthiscommunity, inparticular, fromtheFacultyofSciences,

fortheir invaluableassistance, expertise, andfriendship: Professors

MiguelCorreia, AntnioCasimiro, AlyssonBessani, AnaRespcio,

LusCorreia fromtheDepartmentof Informatics (DI) andProfessor

JooGomes fromtheDepartmentofStatisticsandOperationalRe-

search(DEIO).

A veryspecialandkindthanksgoestoallmypastandpresentcol-

leaguesat theLarge-Scale Informatics SystemsLaboratory (LaSIGE)

researchlaboratory, fortheirfriendshipandtirelesssupport. Inpartic-

ular, thankyouverymuchGiuliana, Mnica, Vinicius, Letcia, Hen-

rique, Andr, Simo, Bruno, Tiago, andMiguel.

Finally, I gratefullyacknowledgethefundingsourcesthatmadethis

doctoralworkpossible. I wassupportedbytheFundaoparaaCin-

ciaeTecnologia(FCT) throughtheDoctoralDegreeGrantSFRH/BD/-

44336/2008, throughtheprojectsPOSC/EIA/61643/2004(AJECT) and

PTDC/EIA-EIA/100894/2008(DIVERSE),andbytheMulti-annualand

CMU-PortugalProgrammes. Inaddition, thisworkwasalsosupported

bytheEuropeanComissionthroughprojectsIST-2004-27513(CRU-

TIAL) andFP7-257475(MASSIF).

Aomeupai.

CONTENTS

ListofFigures xix

ListofTables xxiii

ListofAlgorithmsandListings xxv

ListofAcronyms xxvii

ListofPublications xxxi

1 Introduction 1

1.1 ObjectiveandMainAreasofWork . . . . . . . . . . . . . . . . . 4

1.2 SummaryofContributions . . . . . . . . . . . . . . . . . . . . . . 9

1.3 StructureoftheThesis . . . . . . . . . . . . . . . . . . . . . . . . 11

2 RelatedWork 13

2.1 DetectionofFaultsandVulnerabilities . . . . . . . . . . . . . . . . 14

2.1.1 Faultinjection . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.1.2 Manualanalysisandtestoracle . . . . . . . . . . . . . . . 17

2.1.3 Modelchecking . . . . . . . . . . . . . . . . . . . . . . . . 19

2.1.4 Staticanalysis . . . . . . . . . . . . . . . . . . . . . . . . . 22

xvi CONTENTS

2.1.5 Robustnesstestingandfuzzing . . . . . . . . . . . . . . . . 27

2.1.6 Vulnerabilityscanners . . . . . . . . . . . . . . . . . . . . 29

2.1.7 Run-timedetection . . . . . . . . . . . . . . . . . . . . . . 30

2.1.8 Resourceusagedetection . . . . . . . . . . . . . . . . . . 33

2.2 ProtocolSpecification . . . . . . . . . . . . . . . . . . . . . . . . 35

2.2.1 Manuallydefined . . . . . . . . . . . . . . . . . . . . . . . 36

2.2.2 Dynamicanalysis . . . . . . . . . . . . . . . . . . . . . . . 38

2.2.3 Grammaticalinduction . . . . . . . . . . . . . . . . . . . . 40

2.3 GenerationofTestCases . . . . . . . . . . . . . . . . . . . . . . . 43

2.3.1 Combinatorialtestdatageneration . . . . . . . . . . . . . 44

2.3.2 Randomdatagenerationandfuzzing . . . . . . . . . . . . 45

2.3.3 Dynamicanalysistestdatageneration . . . . . . . . . . . . 47

2.3.4 Symbolicexecutiontestdatageneration . . . . . . . . . . 47

2.3.5 Specification-basedtestdatageneration . . . . . . . . . . . 49

3 NetworkAttackInjectionFramework 53

3.1 GeneralMethodology . . . . . . . . . . . . . . . . . . . . . . . . 54

3.2 OverviewoftheFramework . . . . . . . . . . . . . . . . . . . . . 59

3.3 ProtocolSpecification . . . . . . . . . . . . . . . . . . . . . . . . 61

3.4 AttackGeneration . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

3.5 Injection&Monitoring . . . . . . . . . . . . . . . . . . . . . . . . 64

3.6 AttackAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

3.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4 ProtocolSpecification 71

4.1 CommunicationProtocol . . . . . . . . . . . . . . . . . . . . . . . 72

4.1.1 Protocollanguage . . . . . . . . . . . . . . . . . . . . . . . 74

4.1.2 Protocolstatemachine . . . . . . . . . . . . . . . . . . . . 75

4.2 ManualProtocolSpecification . . . . . . . . . . . . . . . . . . . . 76

4.3 ProtocolReverseEngineering . . . . . . . . . . . . . . . . . . . . 78

4.3.1 Inferringthelanguage . . . . . . . . . . . . . . . . . . . . 81

4.3.2 Inferringtheprotocolstatemachine . . . . . . . . . . . . . 90

4.3.3 Inputversusinput/outputstatemachine . . . . . . . . . . . 95

4.3.4 Experimentalevaluation . . . . . . . . . . . . . . . . . . . 98

CONTENTS xvii

4.4 AutomaticallyComplementingaSpecification . . . . . . . . . . . 111

4.4.1 Overviewoftheapproach . . . . . . . . . . . . . . . . . . 113


4.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

5 AttackGeneration 125

5.1 CombinatorialTestCaseGeneration . . . . . . . . . . . . . . . . . 126

5.1.1 Delimitertestdefinition . . . . . . . . . . . . . . . . . . . 128

5.1.2 Syntaxtestdefinition . . . . . . . . . . . . . . . . . . . . . 131

5.1.3 Valuetestdefinition . . . . . . . . . . . . . . . . . . . . . 132

5.1.4 Privilegedaccessviolationtestdefinition . . . . . . . . . . 135


5.2 Recycling-basedTestCaseGeneration . . . . . . . . . . . . . . . . 137

5.2.1 Overviewoftheapproach . . . . . . . . . . . . . . . . . . 139

5.2.2 Toolimplementation . . . . . . . . . . . . . . . . . . . . . 142


5.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

6 Injection, Monitoring, andAnalysis 165

6.1 Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

6.1.1 Singleinjectioncampaignwithrestart . . . . . . . . . . . . 166

6.1.2 Singleinjectioncampaignwithoutrestart . . . . . . . . . . 167

6.1.3 Repeatedinjectioncampaignwithrestart . . . . . . . . . . 168

6.2 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

6.2.1 Externalmonitor . . . . . . . . . . . . . . . . . . . . . . . 170

6.2.2 Genericinternalmonitor . . . . . . . . . . . . . . . . . . . 171

6.2.3 Specializedinternalmonitor . . . . . . . . . . . . . . . . . 172

6.3 AnalysisoftheAttacks . . . . . . . . . . . . . . . . . . . . . . . . 174

6.3.1 Analysisthroughfaultpatterndetection . . . . . . . . . . . 175

6.3.2 Analysiswitharesourceusageprofile . . . . . . . . . . . . 176

6.3.3 Analysiswithabehavioralprofile . . . . . . . . . . . . . . 183

6.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

xviii CONTENTS

7 AttackInjectionTools 1957.1 AJECT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

7.1.1 Architectureandimplementation . . . . . . . . . . . . . . 1977.1.2 Experimentalevaluation . . . . . . . . . . . . . . . . . . . 199

7.2 PREDATOR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2157.2.1 Architectureandimplementation . . . . . . . . . . . . . . 2167.2.2 Experimentalevaluation . . . . . . . . . . . . . . . . . . . 219

7.3 REVEAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2307.3.1 Architectureandimplementation . . . . . . . . . . . . . . 2307.3.2 Experimentalevaluation . . . . . . . . . . . . . . . . . . . 232

7.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

8 ApplicationsoftheDevelopedTechniques 2458.1 DIVEINTO:SupportingDiversityinIT Systems . . . . . . . . . . . 246

8.1.1 OverviewofanIT system . . . . . . . . . . . . . . . . . . 2488.1.2 Classificationofviolations . . . . . . . . . . . . . . . . . . 2508.1.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 2548.1.4 Toolimplementation . . . . . . . . . . . . . . . . . . . . . 2628.1.5 Experimentalevaluation . . . . . . . . . . . . . . . . . . . 2638.1.6 Case-study . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

8.2 AdaptiveIDS forCriticalServers . . . . . . . . . . . . . . . . . . . 2768.2.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 277

8.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

9 ConclusionsandFutureWork 2839.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2849.2 FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Bibliography 291

Index 311

LIST OF FIGURES

3.1 Theattackprocessonafaulty(orvulnerable)component. . . . . . 55

3.2 Theattackinjectionmethodology. . . . . . . . . . . . . . . . . . . 57

3.3 Frameworkofnetworkattackinjection. . . . . . . . . . . . . . . . 60

4.1 ProtocolinputlanguageoftheFTP. . . . . . . . . . . . . . . . . . 75

4.2 ProtocolstatemachineoftheFTP. . . . . . . . . . . . . . . . . . . 76

4.3 ScreenshotoftheAJECT protocolspecification. . . . . . . . . . . . 77

4.4 ReverX overviewforinferringaninput/outputspecification. . . . . 79

4.5 ExampleofanFTP networktrace. . . . . . . . . . . . . . . . . . . 83

4.6 Inferenceoftheprotocollanguage. . . . . . . . . . . . . . . . . . 84

4.7 Inferredoutputlanguage. . . . . . . . . . . . . . . . . . . . . . . . 90

4.8 Input/outputstatemachineinference. . . . . . . . . . . . . . . . . 93

4.9 Inputandinput/outputstatemachineinference. . . . . . . . . . . 97

4.10 Additionalsessioninthenetworktrace. . . . . . . . . . . . . . . . 98

4.11 Impactofthesamplesizeonthelanguageinference. . . . . . . . 105

4.12 InferredprotocollanguageversusRFC 959. . . . . . . . . . . . . . 107

xx LIST OF FIGURES

4.13 Impactofthesamplesizeonthestatemachineinference. . . . . . 109

4.14 InferredprotocolstatemachineversusRFC 959. . . . . . . . . . . 110

4.15 ReverX averageexecutiontime. . . . . . . . . . . . . . . . . . . . 111

4.16 FTP inputstatemachinecomplemented. . . . . . . . . . . . . . . 122

5.1 Recycling-basedtestcasegenerationapproach. . . . . . . . . . . 139

5.2 Architectureoftheimplementedtool. . . . . . . . . . . . . . . . . 142

5.3 Reverseengineeringtheprotocolspecification. . . . . . . . . . . . 145

5.4 Analysisofthetruevulnerabilitycoverageofexperiment3. . . . . 161

6.1 Rangeofmonitoringsolutions. . . . . . . . . . . . . . . . . . . . . 170

6.2 Specializedinternalmonitor. . . . . . . . . . . . . . . . . . . . . . 172

6.3 SubsetoftheFTP specification. . . . . . . . . . . . . . . . . . . . 185

6.4 SubsetoftheFTP specificationextendedwithmonitoringdata. . . 187

6.5 Obtainingabehavioralprofile(learningphase). . . . . . . . . . . 189

6.6 Usingthebehavioralprofile(testingphase). . . . . . . . . . . . . . 191

7.1 ArchitectureoftheAJECT tool. . . . . . . . . . . . . . . . . . . . . 198

7.2 InputstatemachineofthePOP3protocol. . . . . . . . . . . . . . 200

7.3 InputstatemachineoftheIMAP4Rev1protocol. . . . . . . . . . . 201

7.4 ArchitectureofthePREDATOR tool. . . . . . . . . . . . . . . . . . 217

7.5 DNS messageformat. . . . . . . . . . . . . . . . . . . . . . . . . . 219

7.6 MemoryusageprofilesfortheD1 syntheticserver(withthreadleak).226

7.7 MemoryconsumptioninMaraDNS. . . . . . . . . . . . . . . . . . 229

7.8 ArchitectureoftheREVEAL tool. . . . . . . . . . . . . . . . . . . . 231

7.9 Vulnerabilitydetectionoftestcases(Experiment1). . . . . . . . . 237




LIST OF FIGURES xxi


8.1 IT systemarchitecture. . . . . . . . . . . . . . . . . . . . . . . . . 249

8.2 Methodologyoverview. . . . . . . . . . . . . . . . . . . . . . . . . 255

8.3 Input/outputstatemachineofanexampleprotocol. . . . . . . . . 256

8.4 BehaviormodelforFTP serverIIS6withconformanceviolations

fromIIS5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

8.5 Inferringthebehavioralprofile(learningphase) . . . . . . . . . . . 278

8.6 Usingthebehavioralprofile(operationalphase) . . . . . . . . . . 280

LIST OF TABLES

4.1 Characterizationofthetrainingsets. . . . . . . . . . . . . . . . . . 100

4.2 Evaluationoftheinferredinputandoutputlanguages. . . . . . . . 104

4.3 DiscoveredmessageformatsandrespectiveRFC extensions. . . . 121

5.1 Exampleofasetoftestcasesandtherespectiveextractedpayloads.141

5.2 Targetservertransitiontableandpreludegeneration. . . . . . . . . 147

5.3 Testcasegenerationexample. . . . . . . . . . . . . . . . . . . . . 149

5.4 TestcasegenerationtoolsforFTP protocol. . . . . . . . . . . . . . 151

5.5 CoverageoftheFTP protocolspace. . . . . . . . . . . . . . . . . . 152

5.6 Taxonomyofpayloadsfromexploitsof131FTP vulnerabilities. . . 156

5.7 PotentialvulnerabilitycoverageofFTP testcases(part1). . . . . . 156

5.8 PotentialvulnerabilitycoverageofFTP testcases(part2). . . . . . 157

5.9 Testcasegenerationtoolsfornon-FTP protocols. . . . . . . . . . . 158

5.10 Potentialvulnerabilitycoverageofnon-FTP testcases(part1). . . . 159

5.11 Potentialvulnerabilitycoverageofnon-FTP testcases(part2). . . . 160

xxiv LIST OF TABLES

7.1 TargetPOP andIMAP e-mailservers. . . . . . . . . . . . . . . . . 204

7.2 E-mailserverswithnewlydiscoveredvulnerabilities. . . . . . . . . 209

7.3 Syntheticleakserverswithresourceleaks. . . . . . . . . . . . . . 221

7.4 Projectionsforadiskandmemoryleakcreatedfrom n injections. . 222

7.5 Resourceusageprojectionsforthesyntheticleakservers(with p = 2).224

7.6 R2a andMSE fortheresourceusageprofilesforthesyntheticleak

servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

7.7 ResourceusageprofilesfortheDNS servers. . . . . . . . . . . . . 228

7.8 ReportedknownFTP vulnerabilities. . . . . . . . . . . . . . . . . . 233

8.1 Referenceandtestingtraces. . . . . . . . . . . . . . . . . . . . . . 258

8.2 Correlationtable. . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

8.3 OS andserverconfigurationofthethreeIT scenarios. . . . . . . . 264

8.4 ViolationsdetectedontheFTP replicationscenario. . . . . . . . . 267

8.5 ViolationsdetectedontheSMTP replicationscenario. . . . . . . . 270

8.6 ViolationsdetectedonthePOP replicationscenario. . . . . . . . . 271

LIST OF ALGORITHMS AND LISTINGS

4.1 Generalizationoftheprotocollanguage. . . . . . . . . . . . . . . 874.2 Mergingprocesstoproducetheprotocolinput/outputstatemachine. 944.3 Complementingthelanguageofanexistingprotocolspecification. 1154.4 Complementingthestatemachineofanexistingprotocolspecifi-

cation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1195.1 AlgorithmfortheDelimiterTestgeneration. . . . . . . . . . . . . . 1305.2 AlgorithmfortheSyntaxTestgeneration. . . . . . . . . . . . . . . 1315.3 AlgorithmfortheValueTestgeneration. . . . . . . . . . . . . . . . 1335.4 Algorithmforthegenerationofmaliciousstrings. . . . . . . . . . . 1347.1 FilewithmalicioustokensforPOP protocol. . . . . . . . . . . . . 2067.2 FilewithmalicioustokensforIMAP protocol. . . . . . . . . . . . . 2068.1 DetectedviolationsbetweenIIS6andIIS5. . . . . . . . . . . . . . 2698.2 Excerptofthenormalizer(Python). . . . . . . . . . . . . . . . . . 274

LIST OF ACRONYMS

ABNF AugmentedBackus-NaurForm

AJECT AttackInJECtionTool

API ApplicationProgrammingInterface

ARP AddressResolutionProtocol

ASN.1 AbstractSyntaxNotationOne

AST AbstractSyntaxTree

BNF Backus-NaurForm

COTS CommercialOff-The-Shelf

CR CarriageReturn

CT CommandnamesThreshold

CVE CommonVulnerabilitiesandExposures

xxviii LIST OF ACRONYMS

DIVEINTO DIVErseINtrusionTOlerantsystems

DNS DomainNameSystem

DoS Denial-of-Service

DS DistinguishingSequence

FSM Finite-StateMachine

FTP FileTransferProtocol

GUI GraphicalUserInterface

ID identifier

IDS IntrusionDetectionSystem

IEEE InstituteofElectricalandElectronicsEngineers

IETF InternetEngineeringTaskForce

IMAP InternetMessageAccessProtocol

ISO InternationalOrganizationforStandardization

IT IntrusionTolerance

ITU-T InternationalTelecommunicationUnionTelecommunicationStandardiza-

tionSector

LBL LawrenceBerkeleyNationalLaboratory

LF LineFeed

LTL LinearTemporalLogic

LIST OF ACRONYMS xxix

IP InternetProtocol

MSE MeanSquareError

NP NondeterministicPolynomialtime

OS OperatingSystem

POP PostOfficeProtocol

PREDATOR PREDictingATtacksOnResources

PTA PrefixTreeAcceptor

REVEAL REvealingVulnerabilitieswithbEhAvioralprofiLe

RFC RequestForComments

SDL SpecificationandDescriptionLanguage

SIP SessionInitiationProtocol

SMTP SimpleMailTransferProtocol

SQL StructuredQueryLanguage

SWIFI Software-ImplementedFaultInjection

TCP TransmissionControlProtocol

UIO UniqueInput/Output

VAT VulnerabilityAssessmentTool

VDM-SL ViennaDevelopmentMethodSpecificationLanguage

VM VirtualMachine

xxx LIST OF ACRONYMS

VT VariabilityThreshold

W3C WorldWideWebConsortium

XSS cross-sitescripting

LIST OF PUBLICATIONS

InternationalConferencesandJournals

[P1] Antunes, J.andNeves, N., InferringaProtocolSpecificationfromNetworkTraces. Submittedforpublicationinajournal.

[P2] Antunes, J.andNeves, N., RecyclingTestCasestoDetectSecurityVulner-abilities. Submittedforpublicationinaconference.

[P3] Antunes, J. andNeves, N., UsingBehavioral Profiles toDetect SoftwareFlawsinNetworkServers. InProceedingsoftheInternationalSymposiumonSoftwareReliabilityEngineering(ISSRE),Hiroshima, Japan, November2011.

[P4] Antunes, J., Neves, N., andVerissimo, P., ReverX:ReverseEngineeringofProtocols. InProceedingsoftheWorkingConferenceOnReverseEngineer-ing(WCRE),Limerick, Ireland, October2011.

[P5] Antunes, J. andNeves, N., DiveInto: SupportingDiversity in Intrusion-TolerantSystems. InProceedingsof theInternationalSymposiumonRe-liableDistributedSystems(SRDS),Madrid, Spain, October2011.

[P6] Antunes, J.andNeves, N., AutomaticallyComplementingProtocolSpeci-ficationsFromNetworkTraces. InProceedingsoftheEuropeanWorkshoponDependableComputing(EWDC),Pisa, Italy, May2011.

xxxii LIST OF PUBLICATIONS

[P7] Antunes, J., Neves, N., Correia, M., Verissimo, P., andRuiNeves. Vul-nerabilityDiscoveryWithAttackInjection. IEEE TransactionsonSoftwareEngineering, 36:357-370, May/June2010.

[P8] Antunes, J., Neves, N., andVerissimo, P., DetectionandPredictionofRe-source-ExhaustionVulnerabilities. InProceedingsoftheInternationalSym-posiumonSoftwareReliabilityEngineering(ISSRE),Seattle, WA,USA,No-vember2008.

NationalConferences, FastAbstracts, StudentPapers

[P9] Antunes, J.andNeves, N., AdaptiveMonitoringtoDetectIntrusionsinCrit-icalServers. InFastAbstractinSupplementoftheInternationalConferenceonDependableSystemsandNetworks(DSN),Boston, MA,USA,June2012.

[P10] Antunes, J., Neves, N., andVerissimo, P., UsingAttackInjectiononClosedProtocols. InFastAbstractinSupplementoftheInternationalConferenceonDependableSystemsandNetworks(DSN),Chicago, USA,June2010.

[P11] Antunes, J.andNeves, N., BuildinganAutomatonTowardsProtocolRe-verseEngineering. InProceedingsoftheSimpsiodeInformticaInforum,Lisboa, Portugal, September2009.

[P12] Antunes, J., Neves, N., andVerissimo, P., FindingLocalResourceExhaus-tionVulnerabilities. InStudentpaperat the InternationalSymposiumonSoftware Reliability Engineering (ISSRE),Trollhttan, Sweden, November2007.

[P13] Teixeira, E., Antunes, J., andNeves, N., AvaliaodeFerramentasdeAnliseEsttica de Cdigo para Deteco deVulnerabilidades. In Proceedingsof theSeguranaInformticanasOrganizaes(SINO),Lisboa, Portugal,November2007.

TechnicalReports

[P14] Antunes, J., Neves, N., andVerissimo, P., ReverX:ReverseEngineeringofProtocols. TechnicalReportTR-2011-01, FaculdadedeCinciasdaUni-versidadedeLisboa, January2011.

[P15] Franceschinis, G., Alata, E., Antunes, J., Beitollah, H., Bessani, A., Correia,M., Dantas, W., Deconinck, G., Kaniche, M., Neves, N., Nicomette, V.,

LIST OF PUBLICATIONS xxxiii

Sousa, P., andVerissimo, P., ExperimentalValidationofArchitecturalSolu-tions(CRUTIAL DeliverableD20), TechnicalReportTR-2009-008, Facul-dadedeCinciasdaUniversidadedeLisboa, March2009.

CHAPTER 1

INTRODUCTION

Inrecenthistory, wehavewitnessedagrowingnumberofscientificandtechno-

logicalinnovationsthatarecriticaltotheadvancementofknowledge, theprop-

agationofculture, andthegenerationofwealth. TheInternetwasoneofthose

innovationsandithadaprofoundimpactonsociety, revolutionizingthewaywe

communicateandlive.

ItisestimatedthatthenumberofInternetusersexceeded2.4billionin2011,

morethanthedoubleof2006(ITU, 2011). Inparallelwithitsincreasingpopu-

larity, theInternethasbecomeamajoreconomicdriver, representingatrading

volumeofalmost6trillionEurosworldwideperyear. ItisestimatedthattheWeb

representsonaverageabout3%ofGDP inthedevelopedcountriesorincoun-

trieswithstrongeconomicgrowth, surpassingtheagricultureandenergysectors1

1ThisstudyincludedtheG8countries, theemergingeconomiesofChina, IndiaandBrazil,

2 1. INTRODUCTION

(Rausaset al., 2011).

TheInternetisthereforeseenasanessentialservicetosocietyandeconomy,

generatingjobsandknowledge, changinglifestylesandevenhavingapolitical

impact, suchasintherecentuprisingsintheMiddleEastandNorthAfricasup-

portedbysocialnetworking(Coz, 2011). Thisdependenceisfurtheraggravated

bytheInternetsroleinthemanagementofsomecriticalinfrastructures, without

whichsociety, asweknowit, couldnotsurvive. Transport, communication, en-

ergy, andbanking, aresomeoftheservicesthatwedependonandarecontrolled

andoperatedthroughtheInternet. Thedisruptionoftheseservicescanhaveseri-

ousanddevastatingconsequences(US-CanadaPowerSystemOutageTaskForce,

2006; Castle, 2006), makingtheInternetanextremelysensitivepointoffailure,

bothtophysicalandcyberattacks.

ThesecuritycompromiseofaserverontheInternetmayaffectnotonlyits

users, butcouldalsotriggeracascadeeffectwithseriousrepercussionsonother

services(Sharma, 2010). Severalrecentstudiesandsurveysrevealthatthenumber

ofattacksisescalatingandtheyarebecomingincreasinglymoreefficientandso-

phisticated(DobbinsandMorales, 2011; McAfee, 2011; Symantec, 2012). These

attacksarenolongerplainandharmlessdemonstrationsofpowerbetweenhack-

ers, butareillintendedandperpetratedbycarefullyorganizedandwell-financed

groups. Theirmotivesrangefrompersonalprofitandactivism, toindustrialand

inter-governmentalsabotage, andeventerrorism(Garget al., 2003; Andersonand

Nagaraja, 2009; Krekel, 2009; Kanichet al., 2011; Rashid, 2011).

Consequently, protectingthenetworkserversfrombeingexploitedbecomes

essentialtothecorrectandsafeoperationoftheInternet. Sometimes, however,

andtwocountrieswithhighbroadbandpenetration, SwedenandSouthKorea.

3

securityrelatedtasksaredelegatedtothebackground, inpartbecausetheyare

notperceivedasfundamentalastheinclusionofnewfunctionality. Infact, most

oftheeffortindevelopmenttypicallygoestothecreationofnewfeatures, since

theyarecommerciallymoreappealing. Thismakes, forexample, testingopera-

tionsincreasinglycomplexandexpensive, asthecorrectnessofmorefunctional

requirementsneedstobechecked, togetherwithnon-functionalaspects, suchas

ensuringacceptableperformancelevelsandthesafetyandreliabilityofsystems.

Software testing is thefirst lineofdefense inpreventingattacksbecause it

cansupport thediscoveryandremovalofvulnerabilities. However, searching

forerrorsorflaws inapplicationsandservers isapainstakinganderror-prone

process, traditionallyperformedbyhumans. Consequently, theclassicalmethods

of testinghave invariablyoverlookederrors. On theotherhand, themethods

carriedoutbyattackerstofindsoftwareflawshavebeenprovedquiteeffective,

whichisclearlyvisibleinthegrowingnumberofreportedvulnerabilities(Fossi

et al., 2011).

As a result, newermechanisms to increase the correctness of systems are

needed, inparticularwithregardtosecurityflaws. Thisdoctoralworkcontributes

withanewmethodologyforvulnerabilitydiscovery, inspiredbyatechniqueused

bysomeattackersandcomputersecurityanalysts, inwhichtheypersistentlyex-

perimentdifferentformsofattack, lookingforevidencethatavulnerabilitywas

activated. Ourattackinjectionmethodologyfollowsasimilarapproachbysys-

tematicallygeneratingandinjectingtestcases(orattacks), whilemonitoringand

analyzingthetargetsystem(Neveset al., 2006). Anattackthattriggerssomeunex-

pectedbehaviorprovidesastrongindicationofthepresenceofaflawthatcould

beexploitedbythatattackorbysomevariationofit. Thisattackcanthenbe

4 1. INTRODUCTION

giventothedevelopersasatestcasethatreproducestheanomaly, toassistinthe

removaloftheflaw.

Theattack injectionmethodologycanbeapplied tomost typesofcompo-

nentsthatonemaywishtosearchforvulnerabilities. Inthethesis, wenarrowed

our study tonetwork serversbecause, froma securitypointof view, theyare

probablythemostinterestingclassoftargetsystems. First, theselargeandoften

complexapplicationsaredesignedtosustainlongperiodsofuninterruptedoper-

ationandareusuallyaccessiblethroughtheInternet. Second, anintrusionina

networkserverusuallyhasasignificantimpact, sinceacorruptionontheserver

maycompromisethesecurityofallclients(e.g., iftheadversarygetsarootshell).

Consequently, networkserversareahighlycovetedtargetbymalicioushackers.

1.1 ObjectiveandMainAreasofWork

Themainfocusofthisworkistoprovideatheoreticalandexperimentalframe-

workfortheimplementationandexecutionofattackinjectiononnetworkservers,

withtheaimoffindingvulnerabilitiesinanautomatedandsystematicway. The

thesiscoversseveralaspectsrelatedtothisapproach, includingtheinferenceof

aspecificationoftheprotocolimplementedbytheserver, thegenerationoftest

casesandtheirinjection, andthemonitoringofthesystemtodetectfailures. Nat-

urally, suchabroadtopiccomesintocontactwithvariousfieldsofexpertise, each

withitsownwealthofopenproblemsandtechniques. Therefore, wehadtolimit

thescopeoftheworkandconcentrateonaparticularapproachofattackinjec-

tion, unveilingsomeofitschallengesandbenefits, andofferingoriginalsolutions

toaddresseachstepofthemethodology. Futureworkscanthentaketheseideas

1.1ObjectiveandMainAreasofWork 5

andfurtherexplorethemtodeveloptoolsandmethodologiesthatcansupport

thedevelopmentofmoresecureanddependablecomputersystems.

Intherestofthissection, wegiveanoverviewofthemainareasofinvestiga-

tionthatarecoveredinthethesis.

Attackinjection

Thefundamentalobjectiveofthisresearchisthedesignofamethodologythat

coversallthestepsofautomatedattackinjection. Wedefinethisoverallprocess

inthecontextofanattackinjectionframeworkthatisabletoaccommodatedif-

ferenttechniquesthatcanbeusedduringthediscoveryofsecurityvulnerabilities.

The complete frameworkwas implemented in three attack injection tools,

AJECT, PREDATOR, and REVEAL. AJECT isthearchetypalattackinjectiontool

thatadaptsandextendsthetraditionaltechniquesoffaultinjectiontolookfor

securityvulnerabilities. Thistoolwasusedinseveralexperimentswithnetworks

servers, demonstratingthatitcandiscovernewvulnerabilities. PREDATOR isa

toolthatevolvedfrom AJECT, butisfocusedonacceleratingthesoftwareaging

effectsofnetworkserversandindetectingvulnerabilitiesrelatedtotheinappro-

priateuseofresources. Thiskindofvulnerabilitiesistypicallydifficulttodetect

becauseitseffectsarenotimmediatelyperceived. Infact, virtuallyanynetwork

serverconnectedtotheInternetissusceptibletotheexhaustionofresourceswhen

facedwithanoverwhelmingnumberofrequests(e.g., causedbyaSlashdoteffect

(Adler, 1999)orbyaconcerteddistributednetworkattack(Lauet al., 2000)), and

therefore, itisimportanttounderstandtheextentthataserverisvulnerableto

thisproblem. REVEAL isanotherspecializationof AJECT, whichimplementsan

automatedanalysisapproachfordetectingabnormalbehaviors. Flawscanthen

6 1. INTRODUCTION

bediscoveredduringatestingphaseifanattackcausestheservertoexecuteinan

waythatviolatesapreviouslyinferredbehavioralprofilethatmodelsthecorrect

executionofthetargetsystem.

Automaticinferenceofaprotocolspecification

A networkserverprovidesa servicebyexchangingmessageswith theclients,

andthereforeitisappropriatetomodelaserverbyderivingaspecificationofthe

communicationprotocolthatitimplements. Forinstance, a FileTransferProto-

col (FTP) serverismodeledaftertheprotocolspecifiedbythe InternetEngineering

TaskForce (IETF) inthe RequestForComments (RFC) 959(PostelandReynolds,

1985). Thisspecificationcanthenbeusedinseveralcomponentsoftheframe-

work, suchas in thegenerationofmoreeffective test casesor to support the

thoroughmonitoringoftheserver.

However, publicspecifications, suchasthosepublishedbythe IETF andother

standardizationorganizations, arenotalwaysavailable, suchaswhentheserver

implementsaproprietary(orclosed)protocol. Furthermore, evenwhenthepro-

tocolspecificationisaccessible, manuallytranslatingthedocumentationintoa

formalmodelisatimeconsuminganderror-proneprocess. Thethesispresentsa

novelprotocolreverseengineeringapproachthatusesnetworkmessagestoand

fromtheservertoautomaticallyderiveaspecificationoftheprotocol, capturing

thesyntaxofthemessagesandtheprotocolstatemachine. Thissolutionisbased

ontheproblemofinducinggrammars, anditconsistsinderivingaformalmodel

fromafinitesubsetofexamples.

Theapplicationsofthisworkarenotlimitedtoattackinjection. Onceinferred,

thespecificationmayalsobeusedinothertestingapproaches, suchasconformity

1.1ObjectiveandMainAreasofWork 7

testing(Dahburaet al., 1990), ortocreatemoreeffectivedefensemechanisms,

namelyfirewalls(InghamandForrest, 2006; Penget al., 2007)or IntrusionDetec-

tionSystems (IDSs) (Roesch, 1999; Paxson, 1999; HoaglandandStaniford, 2009).

Laterinthedocument, weexplainhowtheprotocolreverseengineeringmech-

anismcanhelponthedeploymentofdiversereplicatedsystemsandmonitoring

ofcriticalservers.

Automaticgenerationofattacks

Theattacksgeneratedbyatoolshouldexercisetheentirefunctionalityprovided

bythetargetsystemandcoverthemostrelevantinputdomain. Inpractice, it

isunfeasible tocreate testcases forallpossible inputvalues. Although ithas

beenproventhatthereexistsafinitesetoftestcasesthatcanreliablydetermine

whetheragivenprogramiscorrect, obtainingsuchasetisanundecidableprob-

lem(Howden, 1976).

Thethesisproposestwosolutionsforobtainingafinitesetoftestcasesthatare

bothexhaustive(coveringthewholespecification)andcomprehensive(searching

forseveralclassesofvulnerability). Bothsolutionsresorttomaliciouspayloads,

whichhavebeenobservedinmostofthereportednetworkattacks, inorderto

determineifitisviabletocircumventtheprotectionmechanismsimplemented

inthetargetsystem.

Onesolution isbasedonacombinatorialgenerationof testcases, suchas

thepairwisetestingapproach, wheredifferentvaluesconsistingoflegalandma-

liciouspayloadsarecombined. Anotherapproachconsistsinrecyclingexisting

testcases, suchas thosegeneratedbyspecializedsecurity tools, toproducea

manageablenumberofattackswithahighcoverageofdifferentclassesofvul-

8 1. INTRODUCTION

nerabilities.

Monitoringandbehaviorinference

Analyzingtheresultsoftheinjectionofattackstoseekevidenceoftheactivation

ofvulnerabilities, suchaserrorsorsystemfailures, canbeachallengingproblem.

Thethesisproposestwoapproachestoaccomplishthisobjective. Onesolution

involvesexplicitly lookingforcharacteristicsofknownflaws, suchasmemory

accessviolationsorcrashes. Dependingonthetypeoferror(orfailure), itsiden-

tificationcanbequitetrivialorrequireacarefulobservationofthetargetsystem

toidentifymoresubtlesignsofanomalies.

A moresophisticatedsolutionconsistsininferringadescriptionofthecorrect

behaviorofthetargetsystem. Forthispurpose, wedevisedatechniquethatbuilds

abehavioralprofilebyextending theprotocol specificationof the serverwith

informationaboutits internalexecution. Thismechanismisthusabletodraw

acompleteanddetailedpictureoftheexpectedbehaviorofthetargetsystem,

allowingittodeterminewhenavulnerabilityisactivated.

The informationabout theserversbehaviorwhileprocessing theattacks is

providedbyamonitorcomponent, whichcanbeimplementedthroughdifferent

methodswithvaryingabilitiesandrestrictions. Threedistincttypesofmonitors

weredevelopedandstudied: a thoroughmonitor, withdetailedandaccurate

monitoring capabilities, but restricted toUnix-based systems; a universal and

simplemonitor, butwithnosupportforresourceusageorexecutiontracing; and

aremotemonitorthatinfersthestatusoftheserverthroughanadditionalnetwork

connection.

1.2SummaryofContributions 9

Applicationstootherdomains

Finally, someofthesolutionsthatresultedfromthisworkhavealsobeenapplied

tootherareas. Oneresearchpaththatwasfollowedusestheprotocolreverse

engineeringtechniquesasaneffectivewaytocomparethebehaviorofdiverse

replicasin IntrusionTolerance (IT) systems. Thisapproachsupportstheevalua-

tionofthecompatibilityofdifferentimplementationsandconfigurationsofrepli-

cas. A toolimplementingthisapproach, DIVEINTO, wasdevelopedandapplied

to three replicationscenarios. Theexperimentsdemonstrate thatvarioussorts

ofviolations, includingproblemsrelatedtonondeterministicexecution, canbe

revealedbythisapproach.

Anotherpossibleapplicationthatwasbrieflyaddressedistheuseofthebe-

havioralprofileanalysistoimplementamoredetailed IDS forcriticalservers. This

novelapproachdiffersfromprevious IDS solutionsbyprovidingameansofselec-

tivelyinspectingtheexecutionofthemostsensibleprotocoltasks. Suchan IDS

canthusattainagreaterlevelofsecurityandprotection, whilestillmaintaining

agoodoverallthroughputandperformance.

1.2 SummaryofContributions

Thissectionsummarizesthemostimportantcontributionsthatresultedfromthis

work.

10 1. INTRODUCTION

Networkattackinjectionframework

Themaincontributionofthisthesisisanattackinjectionframeworkthatencom-

passesthevariousaspectsrelatedtothegenerationandinjectionofattacks, the

monitoringofthetargetsystem, andtheanalysisofresults. Threeattackinjec-

tiontoolsweredeveloped, AJECT, PREDATOR, and REVEAL, whichimplement

alternativeinjectionandmonitoringapproaches, testcasegenerationalgorithms,

andprovideabasis for theautomatedanalysisof theresults. Thesetoolsand

techniqueshavebeenintegratedandrecentlyreleasedintheopensourceproject

AJECT2. Mostdocumentationandexperimentalresultspertainingtotheattackin-

jectionframeworkwerepublishedinP3, P7, P8, P12, andP15, andtherecently

submittedP2.

Protocolreverseengineering

Communicationprotocolsdeterminehownetworkcomponentsinteractwitheach

other. Theabilitytoderiveaspecificationofaprotocolcanbeusefulinmany

contexts, suchas tosupportdeeperblack-boxtestingtechniquesor thedevel-

opmentofeffectivenetworkdefensemechanisms. Unfortunately, sinceitisvery

hardtoobtainthesespecificationsinanautomaticway, littleworkhasbeendone

inthisareainthepast. Thethesisprovidesasolutionbasedongrammarinduc-

tiontoinferaspecificationofaprotocolfromsamplesofnetworktraces. This

approachwasimplementedinanopensourcetoolcalledReverX 3 anditisused

inseveralinstancesofthethesis. TheresultsofusingReverX showthatitisable

toinferprotocolspecificationswithhighlevelsofprecisionandrecall. Themost2http://code.google.com/p/aject/3http://code.google.com/p/reverx/

http://code.google.com/p/aject/http://code.google.com/p/reverx/

1.3StructureoftheThesis 11

relevantpublicationsare: P4, P6, P11, andtherecentlysubmittedP1.

DiversityforIT systems

Thefinalgoalofattackinjectionistopreventintrusionsbyremovingvulnerabil-

ities. Interestingly, thethesisalsocontributedtotheareasofintrusiontolerance

andintrusiondetection. Inparticular, anewmethodologywascreatedtoassist

theintroductionofdiversityin IT systemsandtoevaluatethecomplianceofdi-

versereplicatedcomponents. A toolimplementingthismethodology, DIVEINTO,

wasusedtoidentifyseveralincompatibilitiesbetweendiverseimplementations

andconfigurationsofreplicas. ThefindingswerepublishedinP5.

1.3 StructureoftheThesis

Thisthesisisorganizedasfollows:

Chapter 2 providesthecontextinwhichthethesisappearsandpresentsrelated

workintheliterature.

Chapter 3 describes theoverallattack injection frameworkand itscompo-

nents, whicharedetailedintherestofthethesis.

Chapter 4 dealswiththeproblemofobtainingaspecificationofthecommu-

nicationprotocolof the target system, which is themeans for the injectionof

attacksandalsotoassistintheidentificationofanomalousbehavior.

Chapter 5 addressesthegenerationofattacksbyusingaspecificationofthe

communicationprotocolandbyresortingtomaliciouspayloadspresentinknown

exploitsandtestcasescreatedbysecuritytools.

Chapter 6 presentsdifferentinjectionstrategiesandmonitoringsolutions, and

12 1. INTRODUCTION

discussestheproblemidentifyinganomaliesinthebehaviorofthetargetsystem.

Chapter 7 introduces three attack injection tools, AJECT, PREDATOR, and

REVEAL, thatimplementandevaluatemanyofthetechniquesdescribedinthe

thesis.

Chapter 8 offersafewapplicationsofsomeofthetechniquesthatresulted

from thiswork. Oneapplication isamethodologyanda tool toevaluate the

compatibilityofdiversereplicasin IT systems. Anotherapplicationisbasedon

theinferenceofthecorrectbehaviorofthetargetsystemtoidentifyintrusionsin

real-time.

ThethesisendswithconclusionsandfutureworkinChapter 9.

CHAPTER 2

RELATED WORK

Thethesisinvestigatesandexploresanewapproachforthediscoveryofvulner-

abilitiesinnetworkservers. Atitscore, itresortstotheinjectionofpreviously

generatedattacksandtotheirrespectiveanalysis. Thereare, however, numer-

ouschallengesrevolvingaroundthisapproachthatmustbeaddressed, suchas

theconstructionofspecificationstoassistthegenerationoftestcases, theinjec-

tionandmonitoringoftheattacks, andtheautomaticidentificationofanomalies.

Consequently, thisworktouchesmanyresearchareasandhasbeeninfluenced

byalargebodyofworkfromthescientificliterature.

Weorganizedtherelatedworkinthreemainareasofinterest. Thefirstsec-

tiondealswiththeunderlyingandfundamentalproblemoffindingfaults, and

inparticular, securityvulnerabilities, anditdiscussesseveraltestingapproaches,

14 2. RELATED WORK

includingtheverificationandvalidationofsoftwareandthemonitoringofap-

plications. Then, wereviewsolutionsconcerningthespecificationofprotocols,

whichisanessentialcomponentinnetworkattackinjection. Finally, welookin

moredetailintotheproblemofgeneratingtestcasesthatareaimedatdiscovering

softwareflaws.

2.1 DetectionofFaultsandVulnerabilities

Inthenextsubsection, weprovideabriefoverviewofthetechniquesusedinthe

injectionoffaultsinsoftware, whichhasstronglyinfluencedattackinjection. Af-

terthat, wereviewsomeofthemethodsusedtodetecterrorsinsoftware, suchas

manuallyinspectingthecodeorautomatingthisprocesswithatestoracle, model

checking, andstaticanalysis. Then, theremainingsubsectionsaredevotedtothe

problemoffindingvulnerabilities, namelyfuzzing, scanners, andmechanisms

thatallowthedetectionofsecurityflawsatrun-timeandbymonitoringthere-

sourceutilization.

2.1.1 Faultinjection

Oneofthecoreconceptsinthisthesisistheinjectionoffaults, whichingen-

eraltermsconsistsinpurposelyintroducingoneormorefaultsinthesystemfor

evaluationandtestingpurposes.

Faultinjection isaclassicalexperimentalapproachfortheverificationoffault

handlingmechanisms(faultremoval)andfortheestimationofvariousparameters

thatcharacterizeanoperationalsystem(faultforecasting), suchasfaultcoverage

anderrorlatency(AvizienisandRennels, 1972; Arlatet al., 1993). Traditionally,

2.1DetectionofFaultsandVulnerabilities 15

faultinjectionhasbeenutilizedtoemulateseveralkindsofhardwareproblems,

rangingfromtransientmemorycorruptiontopermanentstuck-atfaults. Different

methodsandtoolshavebeenproposed, initiallyinjectingfaultsatthehardware

level(logicalorelectricalfaults)(Arlatet al., 1989; Gunnefloet al., 1989)and,

morerecently, atthesoftwarelevel(codeordatacorruption)(Segallet al., 1988;

Kanawatiet al., 1992; Hsuehet al., 1997).

The ideabehind fault injection is toevaluate thesystemsexecution in the

presenceofartificiallyinsertedfaults. Theseinjectedfaultstrytomimicrealis-

tichardwareorsoftwarefaults, andtherefore, itevaluatesthesystemsabilityto

copewithrealfaults. However, physicalinjectiontechniquesbecomeimprac-

ticalinmorecomplexhardwaresystems. Toaddressthisdifficulty, someworks

usehardwaredescriptionmodelsandfunctionalsimulationtools, insteadofthe

actual hardware, to simulate the activationof hardware faults (Choi and Iyer,

1992; Jennet al., 1994; Goswamiet al., 1997; Kanicheet al., 2001). Themain

advantageofresortingtofunctionalsimulationisthatitcanbeusedintheearly

stagesofdesigntopredictthebehaviorofhardwareandsoftwarearchitectures,

withoutanytargetprototypeorspecialevaluationhardware. Thisapproachpro-

videsgreatcontrollabilitytoinjectfaultsincomponentsofthesimulationmodel

ofthetargetsystem. However, theaccuracyoftheresultsdependsonthelevel

ofabstractionofthemodel, whichcanneverbeasrealisticastheactualtarget

system.

Software-ImplementedFault Injection (SWIFI) isa low-costandeasilycon-

trollablealternativetophysicalfaultinjectioninrealtargetsystems. SWIFI isless

expensivebecauseitdoesnotrequirespecialhardwareinstrumentationtoinject

faults. Thisisusuallyachievedbychangingthememoryorregistercontentsbased

16 2. RELATED WORK

onspecifiedfaultmodelsthatemulatetheeffectsofhardwarefaults(Arlatet al.,

1989; Arlatet al., 2003)orbymimickingregularhigh-levelprogrammingerrors

(DuresandMadeira, 2003). Eventhough SWIFI canemulatetheerrorbehavior

causedbymostfaults, itcanonlyinjectfaultsintolocationsthatareaccessibleto

software(e.g., byalteringbitsinprograminstructions, addresspointersordata,

orbychangingentiresequencesofinstructions). Moreover, modifyingthetar-

getsystem(e.g., byaddingspecial instructions, softwaretraps)candisturbthe

observedresults, somethingthatshouldbeminimizedasmuchaspossiblebe-

causeitmightaffecttheevaluation. However, giventhegreatersophisticationof

software-implementedfaults, SWIFI canbeusedtotestmorecomplexsystems,

suchasapplicationsoreven OperatingSystems (OSs). Overtheyears, several

SWIFI toolsweredeveloped, and fewexamplesare: Xception (Carreiraet al.,

1998), FERRARI (Kanawatiet al., 1995), FTAPE (TsaiandIyer, 1995)andGOOFI

(Aidemarket al., 2001).

Faultinjectionwasalsoappliedtoinsertsoftwarebugsinapplications. G-

SWFIT isa sophisticated tool that resorts toa libraryofmutationoperators to

determinehowandwheretoinsertsoftwarefaultsinbinarycode(Duresand

Madeira, 2002, 2006). Thelibraryofmutationswasobtainedinafielddatastudy,

whoseaimwastofindthemostfrequenthigh-levelprogrammingerrorsandtheir

translation to low-level code (Dures andMadeira, 2003). The tool scans the

binaryforspecificlow-levelinstructionpatternsandthenperformsmutationson

thosepatternstoemulatehigh-levelproblems. Thisapproachhastheadvantage

ofnotrequiringaccesstothesourcecode. However, itreliesonthecorrectness

andcompletenessofanup-to-datelibrarythatsupportsthesetofemulatedfaults.

Anevolutionfromthepreviousworkconsistsininjectingvulnerabilities, i.e.,


programmingbugsthatcanbeexploitedbyanadversary. Oneapproachbuilds

onthisideatoevaluatethesecuritymechanismsof IDS andvulnerabilityscan-

ners, byinjectingvulnerabilitiesinWebapplications, andtherespectiveattacks

thatshouldtriggerthem(Fonsecaet al., 2009). Basedonafielddatastudyofthe

mostimportantrealworldWebsecurityflaws, twotypesofvulnerabilitiesareem-

ulated, cross-sitescripting (XSS) and StructuredQueryLanguage (SQL) injection

(FonsecaandVieira, 2008). Bothkindsoffaultsrelyontheexploitationofvari-

ables, andthereforethesourcecodeissearchedforallinputvariablesthataffect

SQL queries. Vulnerabilitiesaretheninsertedthroughspecificcodemutationsat

determinedlocations, eachoneresultinginanewvulnerableversionoftheWeb

application. Foreachvulnerableversion, acollectionofattacksisgeneratedto

activatetheinjectedflaw. Anattackisconsideredsuccessfulifitcanmodifythe

structureofthe SQL query, whichwouldnothappenifthevulnerabilityhadnot

beeninjectedandifthesecuritymechanismunderevaluationhadbeenableto

detertheattack. Albeitthisapproachdoesnottrytouncoverunknownvulner-

abilities, itcanbeusedtoevaluatetheeffectivenessandcorrectnessofdifferent

securityprotectionmechanisms.

2.1.2 Manualanalysisandtestoracle

Themostbasicapproachtotesting, andinparticulartotheanalysisoftheout-

putsfromtheexecutionofthetestcases, istoperformmanualinspection. Au-

tomationisusuallylimitedtotheexecutionofthetestcases, suchaswhentests

arerunovernighttopresenttheresultsinthemorning. Sometestingtoolsthen

presentlargeamountsofdatathathavetobeinterpreted. However, theintricate

andcomplexnatureofthetestresultsrequiresadetailedknowledgeaboutthe

18 2. RELATED WORK

system, whichisusuallyabsentinautomatedtesttools. Therefore, thefinalas-

sessmentoftheresultshasbeenlargelydelegatedtohumantesters. Infact, some

empiricalresultsshowthatmostcompaniesdolittleautomatedtesting(Ander-

ssonandRuneson, 2002)andsomewritersevenadvocatethemoderateuseof

testautomationgivenitscost/valueratio(FewsterandGraham, 1999).

Theanalysisofthetestcaseexecutionconsistsinobservingitsoutputtode-

terminethecorrectnessoftheprogramorsystem. An oracle isamechanismthat

canreliablydeterminewhethertheoutputofatestcaseisvalid(Howden, 1978).

Toascertainifatestpassesorfails, theoraclecomparestheoutcomeofthetest

casewiththecorrectandexpectedoutputs. Findingthecorrectoutputsisknown

inthescientificcommunityastheoracleproblem(AmmannandOffutt, 2008).

A completeoracle isabletoprovidecorrectoutputsforanytestcase. One

possiblesolutiontobuildacompleteoracleistoresorttotheprogramspecifica-

tiontoextracttheexpectedresultsintoadatabaseandmanuallyperformlookups

(AmmannandOffutt, 2008). However, thenumberofexpectedoutputscanbe

verylargeandrenderthisapproachunfeasible.

Someauthorsproposewritingahigh-levelfunctional specification, withlogi-

calconstraintsthatmustbesatisfied, asanimplicitcompleteoracle(Bousquetet

al., 1998). Thisspecificationcanbeconstructedfromformaldescriptions, suchas

softwareenvironmentconstraints, functionalandsafety-orientedproperties, soft-

wareoperationalprofilesandsoftwarebehaviorpatterns. Oraclescanalsobe

derivedfromspecificationlanguages(Hall, 1991; StocksandCarrington, 1996).

However, themainchallengeistoprovideareliableandautomatedmeansof

generatingexpectedoutput.

Completeoraclescanbeexpensiveandevenimpossible, andmanualoracles


arecostlyandunreliable. So, someefforthasbeenputtoautomatetheprocess

of producing incompleteoracles. Researchershave suggested that automated

oraclesrequireasimulatedmodelofsystem. Onesimplesolutionemploysatleast

oneadditionalversionof theprogram thatbehavescorrectly (Weyuker, 1982;

ManolacheandKourie, 2001). Thisgoldenversion, whichmustimplementthe

samefunctionalityastheprogramundertest, isthenrunwiththesameinputs

andtheoutputsarecompared.

Anotherapproachmodelstheexpectedbehavioroftheprogramusinga deci-

siontable (Di Luccaet al., 2002). A decisiontableisasuitablewayofpresenting

thecombinationofconditionsthataffecttheprogramexecutionandtherespec-

tiveoutput. Othersolutionsresorttoheuristicapproaches, suchasAI planning

(Memonet al., 2000)orneuralnetworks(Vanmaliet al., 2002).

2.1.3 Modelchecking

Modelchecking canbeusedtoverifyifahigh-levelrequirementisviolatedorto

provethatthesystemsatisfiessomeproperty. A formalmodelthatrepresentsthe

systemunderevaluationisdefinedbyanabstractspecification, whichsimplifies

manydetailsoftheactualimplementation. Then, thereachablestatespaceof

theformalmodelisexhaustivelyexplored, searchingforviolationsofpreviously

definedproperties(Clarkeet al., 1994, 2000). Thethoroughnessatexploringthe

statespaceofthesystemmakesmodelcheckingagoodmethodforfindingerrors

inunusualsystemstates. However, itrequirestheexplicitdefinitionoftheprop-

ertiesbeingchecked. SomeexamplemodelcheckingtoolsareMOPS (Chenand

Wagner, 2002), CMC (Musuvathiet al., 2002), andFiSC (Yanget al., 2006).

MOPS isastaticanalysistoolthatchecksifaprogramcanviolatespecifiedse-

20 2. RELATED WORK

curityproperties(ChenandWagner, 2002; Chenet al., 2004). Securityproperties

aremodeledbyfinitestateautomataandsuppliedtothetool. A securityproperty

mightdefine, forinstance, thata setuid-root programmustdroprootprivilegesbe-

foreexecutinganuntrustedprogram. Otherexamplesofsuchpropertiesinclude:

creating chroot jailssecurely, avoidingraceconditionswhenaccessingthefile

systemorwhencreatingtemporaryfiles, andpreventingattacksonstandardfile

descriptors(e.g., standardinput). MOPS exhaustivelysearchesthecontrol-flow

graphof theprogramtocheck ifanypathmayviolateasafetyproperty. Any

violationisreportedalongwiththeexecutionpaththatcausedit. Insomeexper-

iments, MOPS wasabletofinderrorsinknownnetworkservers, suchasApache,

Bind, OpenSSH,PostFix, Samba, andSendMail.

CMC isamodelcheckerforC andC++thatexecutesthecodedirectly, and

therefore, itdoesnot requirea separatehigh-level specificationof the system

(Musuvathiet al., 2002). CMC emulatesavirtualmachineoran OS. Thesys-

temismodeledasacollectionofinteractingconcurrentprocesses, whereeach

processrunsunmodifiedC orC++codefromtheactualimplementation, which

is scheduledandexecutedbyCMC.Different schedulingdecisionsandother

nondeterministiceventsgivetheabilitytosearchthepossiblesystemstatesfor

violationstothecorrectnessproperties(e.g., theprogrammustnotaccessillegal

memoryoraparticularfunctionshouldnotreturnaninvalidobject). Oneofthe

drawbacksofthisapproachisthattheuserisrequiredtospecifythecorrectness

properties. Inaddition, theuserisalsoresponsibleforbuildingatestenviron-

mentthatadequatelymodelsthebehaviorofthesystemwheretheprogramis

executedthis testenvironment fakesan OS andanythingoutside thesystem

underevaluation.


Lateron, CMC wasusedtocreateamodelcheckinginfrastructureforfilesys-

temscalledFiSC (Yanget al., 2006). ThistoolactuallyrunsaLinuxkernelinCMC.

Themodelcheckerstartswitha formatteddisk, andrecursivelygeneratesand

checkssuccessivestatesbyexecutingstatetransitionsbasedonactionsinduced

byafilesystemtestdriver. Examplesoftheseactionsare: creating, removing,

orrenamingfiles, directories, andhardlinks; writingtoandtruncatingfiles; or

mountingandunmountingthefilesystem. Aseachnewstateisgenerated, FiSC

interceptsalldiskwrites, checkingifthediskhasreachedastatethatcannotbe

repaired(i.e., invalidfilesystem).

High-levelsoftwarerequirementscanbetranslatedinto LinearTemporalLogic

(LTL) properties, whicharethenusedasaspecificationoftheformalmodel(Tan

et al., 2004; Whalenet al., 2006). A modelcheckerthenlooksinallstatesof

themodelforpossibleexecutionsthatviolatetheproperties, generatingthere-

spectivetestcases, i.e., counterexamplesthatillustratehowtheviolationofthe

propertycantakeplace. Thetestcasesarethenusedintheactualprogramto

verifyifthereisactuallyaproblem.

Modelcheckinghasaninherentlimitationwhensecurityvulnerabilitiesare

concerned. Since thecauses foravulnerabilityarenotknown inadvance, it

ishardtoexplicitlyspecifyallsecuritypropertiesinthesoftwarerequirements,

makingtheresultingfunctionaltestcasesunsuitabletodetectthem. Furthermore,

modelcheckingrequiresaformalmodeltobemanuallycreatedorderivedfrom

thesourcecode. Eitherapproachispronetogenerateincompletespecifications,

thusresultinginformalmodelsthatmightnotbeusefulforsoftwareverification.

22 2. RELATED WORK

2.1.4 Staticanalysis

Staticvulnerabilityanalyzers searchthesourcecodeoftheapplicationsforwell-

knowndangerousconstructsandreporttheirfindings(ChessandMcGraw, 2004).

Theprogrammerthengoesthroughthepartsofthecodeforwhichtherewere

warningstodeterminewhethertheproblemactuallyexists. Thisideahasalso

beenextendedtotheanalysisofbinarycode(DuresandMadeira, 2005).

Mostoftheworkinthisareahasfocusedonfindingbufferoverflowvulner-

abilities, althoughithasalsobeenappliedtootherkindsofflaws, suchasrace

conditionsduringtheaccessof(temporary)files(BishopandDilger, 1996). Some

ofthetoolsexaminethesourcecodeforafixedsetofunsafepatterns, orrules,

andthenprovidealistingoftheir locations(Wagneret al., 2000; Viegaet al.,

2000; LarochelleandEvans, 2001; HaughandBishop, 2003).

Lexicalanalysis isoneof thesimplest formsof staticcodechecking. Lexi-

calanalyzersusually look forunsafe library functionsorsystemcalls, suchas

glibcsfunctions gets() or strcpy(). Thesourcecodeisfedtotheanalyzer, which

parsesthecodeintotokensthatarethenmatchedagainstadatabaseofdanger-

ousconstructs(Viegaet al., 2000; Wheeler, 2007; FortifySoftware, Inc. 2009).

Thismethodologyisquiteeffectiveatlocatingprogrammingerrors, however, the

toolwillneverfindparticularproblemsifthecorrespondingpatternhasnotbeen

written. Additionally, these toolshave the limitationofproducingmany false

positives, i.e., warningsthatshouldnotberaised, becausethewayinwhichthe

suspiciouspatternisusedisnotactuallyintroducingavulnerability.

Asanexample, ITS4isalexicalanalysistoolforscanningC andC++code

forvulnerabilities(Viegaet al., 2000). Thissimpletooltriestoautomatealotof


themanualsourcecodeinspectionwhenperformingsecurityaudits. ITS4usesa

databaseofdangerouspatterns, suchasfunctioncallssusceptibletobufferover-

flows, andparsesthesourcecodeintolexicaltokens. Thesetokensarethencom-

paredagainstthepatternsinthedatabase. Anythinginthedatabaseisflagged,

possiblyresultinginalargenumberoffalsepositives/warnings.

FindBugs is a lexical analyzer tool that looks forusual codingmistakes in

Javaprograms (Ayewahet al., 2008). FindBugs isable to recognizenumerous

programmingerrorsanddubiouscodingidiomsbyusingsimpleanalysistech-

niques. Moreover, italsohelpstoidentifyotherdifficulttypesofflaws, suchas

nullpointerdereferencing, whichrequiremorespecificsolutions.

Staticanalysiscanbeimprovedbymimickingsomeofthefeaturesthatmake

someprogramminglanguagesmoresecure. Oneofthesefeaturesisstrong type

checking (usedinJava, forinstance), whichunfortunatelysomeprogramminglan-

guageslack. TheC language, forinstance, wasdesignedwithspaceandperfor-

manceconsiderationsinmind, inanepochwheresecuritywasnotabigconcern.

A C programmercandirectlymanipulatememorypointers, butheisentrusted

withtheresponsibilityofperformingtheboundarychecks, whichissometimes

neglected. Staticanalysiscanimposestrongertypecheckingbyemployingspe-

cialannotations(e.g., typequalifiers)thatarewrittenascommentsinthesource

code(Evanset al., 1994; Fosteret al., 1999;Wagneret al., 2000). Theseadditional

keywordsareignoredbythecompiler, butarerecognizedbythestaticanalyzer

togiveanindicationaboutthedatatypeanddomainofthevariables. Thisin-

formationcanbeusedtoenforcethecorrectusageofthevariablesaccordingto

theirspecifiedtype, suchaspreventingpositiveintegervariablestooverflowto

negativevalues.

24 2. RELATED WORK

A relatedapproachresortstoanadditionalabstractdatatypetocomplement

thebufferdefinition inC inorder todetectbufferoverflows. Memoryalloca-

tionsarecharacterizedaspairsofintegerranges(specifyingtheirreservedspace)

regardlessoftheircontents(Wagneret al., 2000). Thisideawasincorporatedin

BOON,whichformulatesthebufferoverrundetectionproblemasanintegercon-

straintproblem. Thetoolusessimplegraphtheoretictechniquestoconstructan

efficientalgorithmforsolvingintegerconstraints. Usingthisapproach, detecting

bufferoverrunsisaquestionoftrackingtheintegerrangesoftheabstractdata

type. First, aconstraintlanguageisusedtomodelstringoperations, andthen,

integerrangeanalysissolvestheconstraintsystem. Anyconstraintviolationindi-

catesapossiblevulnerability.

CQualisaframeworkforaddingtypequalifierstoaprogramminglanguage

(Fosteret al., 1999). Theuseoftypequalifiers(suchastheconstruct dynamic

nonzero int)supportstypecheckingandtheinferenceofqualifiedtypesand

theirrelationships. Thisframeworkcanthusprovidemoresophisticatedparadigms,

suchaspolymorphism. Inaddition, CQualalsosupports data-flowanalysis by

addingtaginformationtosomeparticulartypesofdata(e.g., externalinputdata)

and tracking theirexecutionpaths (Shankaret al., 2001). Type inferencerules

arethenconstructedtodetectinconsistenciesonhowthedataisactuallyused.

CQualusesthisanalysistodetectdatacomingfromtheoutsidethatisinterpreted

asaformatstringinafunctioncall, thuswarningforpotentialformatstringvul-

nerabilities. The implementationofdata-flowanalysis isbasedonPerls taint

mode(Birznieks, 1998)typesaremarkedas tainted iforiginatingormodified

fromtheoutside. Thetoolthentracksthepropagationofthetainteddata. Ifthere

isanexecutionpathinwhichtainteddataispassedtoafunctionthatexpects


untainteddataoriftainteddataisinterpretedasaformatstring, CQualraisesan

error.

LCLintstartedasanannotation-assistedstaticcheckingtoolforfindingbuffer

overflowvulnerabilities(Evanset al., 1994). Lateron, moreexpressiveannota-

tionswere added, allowing programmers to explicitly state function pre- and

post-conditions (Larochelle and Evans, 2001). The annotations describe some

assumptionsabout thebuffers thatarepassed to functionsand theirexpected

stateasfunctionsreturn. Forinstance, theprogrammercanannotatethefunction

torestrict themaximumsizeofaparticularbuffertoaglobalvariableusedin

thecode, ortospecifytheminimumandmaximumbufferindicesthatcanbe

read. LCLintcombinestraditionaldata-flowanalysiswithconstraintgeneration

andresolution. ThetoolgeneratesconstraintsfortheC statementsbybuildingan

AbstractSyntaxTree (AST) andstoringeachconstraintinthecorrespondingnode.

Then, asthetreeistraversed, constrain-basedanalysistechniquesareemployed

toresolveandcheckthoseconstrains. LCLinttakesintoaccountthevalueofthe

predicatesondifferentcodepathsinordertodetectvulnerabilities. Thereare,

however, certainprogrammingconstructsthatLCLintisunabletointerpret, and

inaddition, manyspuriouswarningsmightendupbeinggenerated.

Anothersolutiontoprovideamoreelaborateanalysisistoexaminetheap-

plicationsflowofcontrol. Thisapproachgenerallyconsistsinparsingthesource

codeandbuildingan AST inordertostudythevariousrelationsamongthediffer-

entmodulesandfunctionsoftheprogram(Bushet al., 2000; Grammatech, 2012).

Theanalyzertraversestheentiretreetosimulatedifferentcontrol-flowpathsand

toidentifyanyinconsistencies(i.e., potentialvulnerabilities). Thisanalysismay

detectproblemslikeinvalidpointerreferences, theuseofuninitializedmemory,

26 2. RELATED WORK

orimproperoperationsonsystemresources(e.g., tryingtocloseanalreadyclosed

filedescriptor).

PREfixisanerrordetectiontoolforC andC++thatperformscontrol-flowanal-

ysisbysimulatingtheexecutionofindividualfunctions(Bushet al., 2000). The

toolautomaticallygeneratesmodelsofthefunctionsthatdescribetheirbehavior

asasetofconditionals, consistencyrules, andexpressionevaluations. It then

tracesindividualexecutionpaths, andwheneverafunctioncallisencountered,

thecorrespondingmodelisusedtosimulatetheactionofeachoperation. By

trackingthestateofthememoryduringthepathexecution, andapplyingcon-

sistencyrulesofthelanguagetoeachoperation, inconsistenciescanbedetected

andreported. Additionally, thedetailed tracking informationof thepathsand

valuescanshedsomelightontheconditionsinwhichsuchinconsistencieshave

manifested. TheprogrammingflawsthatPREfixcanwarnarerelatedtomemory

operations, suchastheuseofuninitializedmemory, dereferencinguninitialized,

null, orinvalidpointers, andmemoryleaks.

Morerecently, otherstaticanalysisapproacheshaveemergedthatcombine

morethanonetypeofanalysis. BEST resortstocontrol-anddata-flowanalysisto

detectsecurityproblemsinbinaries(Wang, 2010). Theflowgraphsareretrieved

bythird-partytools, suchasIDA Pro(Hex-Rays, 2012)orObjDump(GNUBinu-

tils, 2012). Thegraphsarecombinedwithatranslationofthebinarytocreatea

representationoftheprogramresemblingC withanembeddedassemblersyntax,

whichismucheasiertounderstandandtoassess. Securityanalystscancheckthe

binaryforsecurityflawsbyevaluatingtheflowgraphs(suchastoshownesting

relationshipsamongthecontrolstructures)andthegeneratedprogramrepresen-

tation.


Anothersolutionproducesacontrol-flowsecuritymodelinstead, whichcan

beapplied to sourcecodeorbinaries, to identifyexecutionpaths thatviolate

previouslydefinedsecurityproperties(Chunleiet al., 2009). Thisapproachisable

todetectflawsthatallowthememorytobeoverwrittenortheexecutionpathof

thebinary tobehijacked. However, vulnerabilitiesoriginating fromdynamic

linkedlibrariescannotbedetectedbecausetheselibrariesarenotsupportedin

theanalysis.

2.1.5 Robustnesstestingandfuzzing

Robustnesstesting studiesthebehaviorofthesoftwarecomponentsinthepres-

enceoferroneousinputconditions. Inthisapproach, faultsareinsertedatthe

interactionbetweenthecomponentandtheoutside. Typically, the Application

ProgrammingInterface (API) ofthecomponentundertestissuccessivelycalled

withacombinationofcorrectanderroneous(e.g., outofbounds)parameters.

Then, thebehaviorofthecomponentisobservedtodetermineifitcancopewith

baddata, forexamplewithouthalting. Overtheyears, thisapproachhasbeen

appliedtovariousareas, suchasPOSIX anddevicedriver APIs (Koopmanand

DeVale, 1999; Albinetet al., 2004; MendonaandNeves, 2007).

Fuzzing isaspecialcaseofrobustnesstesting. Thistechniquewasinspiredby

theeffectofnoisydial-uplinesthatsometimesscrambledcommandlinechar-

actersandcrashedapplications. Researchersweresurprisedtofindthat these

spuriouscharactersalonewerecausingsuchproblemsinasignificantnumberof

basic OS utilitiessimpleandmatureprograms, subjecttosomeyearsofusage

andtesting, andthereforeregardedasrobustapplications. Researcherswereable

toreproducethesamebehaviorwithFuzzbyfeedingseveralUnixcommandline

28 2. RELATED WORK

utilitieswithrandomcharacters(Milleret al., 1990).

Initssimplestform, fuzzingmakesfewornoassumptionsaboutthesystem

under testbesides the typeof interface (e.g., file format, protocolmessage, or

anenvironmentvariable), anditcanthusbeappliedtoawiderangeofsystems

(Oehlert, 2005; Suttonet al., 2007). Typically, a fuzzerproducesavery large

numberofinputsequencesthatitpresentstotheinterfaceofthetargetsystem.

Sinceitnormallylacksamonitoringmechanismtodetectthefaults, atestcaseis

consideredtofailifthesystemcrashes. Therefore, fuzzingisquitesuccessfulat

detectingvulnerabilitiesthatcausefatalerrors(e.g., bufferoverflowsor Denial-of-

Service (DoS) vulnerabilities), butismorelimitedinsecurityproblemsthatdonot

resultinsuchevidentbehavior(e.g., privilegedaccessviolation, SQL injection,

orsoftwareagingproblems).

FuzztestinggainedsignificantinterestduetoeventslikeMonthoftheBrowser

Bugs, wherenewexploitswereproducedeachdayforthemostpopularInter-

netbrowsers(Moore, 2006). A newgenerationoffuzzersresortedtospecialized

knowledgetogobeyondsimplerandomtestingandtoperformmorecomplexin-

teractionswiththetarget, suchassendingHTTP messages. Thesemessagesstill

complywiththeformatacceptedbytheparsingmechanismsofthetargetsys-

tembutalsocontainirregularfuzzingelementsthat, ifnotproperlyhandled, can

causeerroneousbehaviors(Sutton, 2005; Biege, 20022011; Betouin, 2006; In-

figoInformationSecurity, 2012; AutoSecTools, 2012; NavarreteandHernandez,

2012; Codenomicon, 2012).

Fuzzingframeworks havealsobeendevelopedtoeasetheprocessoftailoring

fuzzing tools to specific systems (Roning, J. et al. 19992003; Greene, 2005;

Rapid7, 2012; AutoSecTools, 2012; NavarreteandHernandez, 2012). Theyallow


thecustomizationofthefuzzingprocessforaparticulartarget, usingforexample

the Backus-NaurForm (BNF) dialectforprotocolspecification(Roning, J.etal.

19992003)andscriptinglanguagesforfileformatgeneration(Greene, 2005)or

thecreationofnewtestingmodules(Rapid7, 2012). Metasploitisoneofthese

fuzzingframeworksthatsupportsthedefinitionofthecompletetestingprocess,

fromthepayloadgenerationtotheactualtestcaseexecution.

Fuzzing, however, canonlygeneratenegativetestcases, i.e., teststhatexplore

situationsnotdefinedinthestandardspecifications, andnormallytheyonlypro-

videa randomsampleof thesystemsbehavior. Additionally, theiranalysis is

usuallytoosuperficialtoallowthedetectionofotherthansimplecrashfailures.

Nevertheless, bugsdetectedbythisapproachareoftenfoundtobeexploitable

vulnerabilities, soitisparticularusefulforsecurityassessment.

2.1.6 Vulnerabilityscanners

VulnerabilityAssessmentTools (VATs) allowsystemadministratorstocheckappli-

cations, computersystems, orevenentirenetworksagainstadatabaseofknown

vulnerabilitiesandmisconfigurations(IBMInternetSecuritySystems, 2006;McAfee,

Inc. 20032012; SaintCorp. 2012; Qualys, Inc. 20082012; TenableNetworkSe-

curity, 200212a; GreenboneNetworksGMBH, 2012; Rapid7, 2012). Thescan-

nerfirst interactswith the target toobtain informationabout itsexecutionen-

vironment(e.g., OS andavailableservices). Thisinformationisthencorrelated

withthedatastoredinthedatabasetodeterminethepotentialproblemsthatthis

typeofsystemknowntocontain. Inaddition, foreachpotentialvulnerability,

thedatabaseoffersatestcasetodetectit. Thetestcasecanbeasthoroughas

arealexploitorassimpleasacheckonthewelcomebanneroftheserver. To

30 2. RELATED WORK

completetheprocedure, thescannercarriesouttherelevanttestcasesforthat

targetsystemandpresentstheresults.

Somescannersalsoattempttobelessintrusive. TenablePassiveVulnerabil-

ityScanner(TenableNetworkSecurity, 200212b), forinstance, passivelyscans

thenetworkforvulnerablesystems, watchingforpotentialapplicationcompro-

mises, clientandservertrustrelationships, andnetworkprotocolsinuse. This

typeof VAT continuouslylistenstonetworktraffic, identifying OSs andservices

byfingerprintingpacketsandcross-referencingthemwithportnumbers.

Eventhough VATs areextremelyusefultoimprovethesecurityofsystemsin

production, theyhavethelimitationofbeingunabletouncovernewvulnerabili-

tiesanyvulnerabilitynotpreviouslyreportedandincorporatedinthedatabase

willremainundetected.

2.1.7 Run-timedetection

Run-timepreventionmechanismschangetheprogramsexecutionenvironment

withtheobjectiveofdiscoveringandthwartingtheongoingexploitationofvul-

nerabilities. Theideahereisthatremovingallbugsfromaprogramisinfeasible,

andthereforeitispreferableto contain thedamagecausedbytheirexploitation.

Insteadofsolvingtheproblematthesource(i.e., bycorrectingtheprogram), they

trytomenditattheend(e.g., bypreventingthestackfrombeingoverflowed).

Althoughthesemechanismsareusuallyimplementedatcompile-time, thedetec-

tionandpreventionisachievedatthetimeofexecution.

Most of thesemechanismswere developed to protect systems frombuffer

overflows. Attackersexploitthesevulnerabilitiesbysupplyingspeciallycrafted

datatohijacktheprogramsflowofcontrol. Forexample, byoverflowingafunc-


tionsreturnaddresswithoneprovidedbytheattacker, theprogramwillexecute

the instructions locatedat thataddress, whichcan, for instance, spawnaroot

shell. Intheliterature, thereareseveralexamplesoftoolsdesignedtoprotectthe

integrityofthestack, oratleasttodetectanyviolation.

StackGuardandProPolicearecompilerextensions thatprovide themeans

todetect invalidchanges to the functions returnaddressandtoprevent those

changes fromcorrupting theprograms execution (Cowanet al., 1998; Wagle

andCowan, 2003; Etoh andYoda, 2002). Stacksmashingattacks typicallyex-

plorethefactthatthereturnaddressislocatednearalocalvariablewithweak

boundschecking. Therefore, theattackeronlyhastooverwritethememoryfrom

thatvariablesaddresstothereturnaddress. Byplacinga canaryword (i.e., a

pre-selectedrandomvalue)nexttothereturnaddressofafunction, StackGuard

candetectbufferoverflowsonthestack. First, itchecksifthecanarywordisin-

tactbeforejumpingtotheaddresspointedbythereturnaddress, i.e., before the

functionreturns. Ifachangeisfound, thisprobablymeansthatthereturnaddress

isalsomodified, andtheexecutionoftheprogramisaborted(i.e., fail-safestop).

Otherauthorsextendthe ideaofusingcanaries toprotectnotonly there-

turnaddress, butalsootherstackareassusceptibletooverflow, thiswayaiming

atpreventingalltypesofstackoverflow(Zquete, 2004). Eachlocalvariableis

precededbyacanaryword. Theseboundarycanariesarestoredintheformof

alinkedlist, whereeachentryisprotectedusingthepreviousXOR canary. This

wayanattackercausinganoverflowofastackvariablecannoteasilyguessthe

validvaluesoftheboundarycanariesoftheneighboringvariables. Inaddition,

twomodesofrun-timevalidationareprovidedforeitherdevelopmentorproduc-

tionscenarios.

32 2. RELATED WORK

These toolscanalsobeconfigured toprevent returnaddressmodifications

fromoccurringwiththeaimofavoidingtheinterruptionoftheprogramsexecu-

tion. Byresortingtofinegrainmemoryprotection, atoolcanrestrictaccesstoin-

dividualmemoryaddressesviaaspecial API, asprovidedbyMemGuard(Cowan

et al., 1997). Thereturnaddressissetasread-onlymemorywhenthefunctionis

called, andtheprotectionisonlyraisedwhenthefunctionreturns. Attackscan

thusbepreventedbecausetheonlywaytooverwritethereturnaddresswouldbe

throughtheMemGuard API.

More sophisticated techniquesmitigatepointer corruption exploits. Point-

Guard(Cowanet al., 2003)addsspecialcodetotheprogramthatpreventsan

attackerfromproducingpredictablepointervalues. A keygeneratedatrun-time

isusedtoencryptpointerswhiletheyareinmemory. Whenapointerisderefer-

enced, itsvalueisdecryptedfrommemoryandloadedintotheCPU register. Ifan

attackeroverwritesthepointervalue, theprogramwilljumpintoanunpredictable

memoryaddress, thusmakingtheattackimpracticable.

A fewstudiescomparetheeffectivenessofsomeof

network attack injection - di.fc.ul.ptnuno/thesis/joaoantunes_phd_maio12.pdf · analyzing the...

Documents