taming false alarms from a domain-unaware c analyzer by a statistical post analysis

Taming False Alarms from a Domain-Unaware

C Analyzerby a Statistical Post

Analysis

Yungbum Jung, Jaehwang Kim,Jaeho Shin, Kwangkeun Yi

Programming Research Lab.Seoul National University

Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005

Jaeho Shin 2

Motivation: an Industry’s Challenge

In 2004, a company’s SQA dept. asked us for a C buffer-overrun static analyzer that must be sound must have a reasonable cost must be domain-unaware

Our path Sound analyzer: drive cost-accuracy balance to

a limit Statistical filter: sift out inevitable false alarms

and rank alarms by their true probabilities


Jaeho Shin 3

Outline

Airac, Our Analyzer Internals Performance

Statistical Analysis Symptoms Models

Bayesian Analysis Linear Logistic Regression

Sifting out, Ranking


Jaeho Shin 4

Airac

Array Index Range Analyzer for C Our static analyzer

Is an abstract interpreter Does numerical interval analysis Is sound

in sense of detecting all possible buffer overruns

Covers full ANSI C + some GNU extensions


Jaeho Shin 5

Abstraction

Usual abstraction for stateful programs

Set of concrete machine transition

traces

Map from program points to abstract statesPgmPt State

α


Jaeho Shin 6

Abstract Domains

Machine = State x PgmPt State = Stk x Mem x Dmp Mem = Addr Val Val = Interval x 2Addr x 2Array

Addr = PgmVar + AllocSite + AllocSite x Field Array = AllocSite x Base x Size AllocSite = PgmPt [a, b] ∈ Interval = Base = Size

...


Jaeho Shin 7

Techniques Used

Accuracy improvement by narrowing after widening flow-sensitivity context pruning (limited to linear expressions) static inlining (parameterized) static loop unrolling (parameterized)

Cost reduction by careful worklist order: lazy at join points selective join/compare stack obviation


Jaeho Shin 8

Stack Obviation

Size of Stk proportional to program size Most of the analysis time = join + compare OK to skip join/compare for Stk

if changes of Stk always reflected on Mem By simple syntactic transformation

e1 ? e2 : e3 { if (e1) t = e2 else t = e3; t } e[f()] t = f(); e[t]

3~5 times speed up


Jaeho Shin 9

Optimistic Assumption:

i [0, 9] j [0, 18]

Error Recovery During Analysis

1: int a[10], i, j;

2: for (i=0;i<10;i++) {

3: a[i] =2 * i;

4: }

5: j = a[i];

6: a[i] = …

…

buffer overrunsince i [10, 10]


Jaeho Shin 10

Warnings about Performance

Assume typeful C programs arrays must be used as the same type declared

Artificial semantics after errors e.g. overrun, null dereference

No side-effect for library functions No main() then

analyze procedures in their defined order No alarms about buffers whose size is top Top value for free variables


Jaeho Shin 11

Performance 1/2

Linux kernel 2.6.4 Alarms Real Errors LOC Time (sec)

vmax302.c (79)

1 1 246 3

xfrm_user.c (235) 2 1 1,201 109

usb-midi.c (332) 10 4 2,206 3617

atkbd.c (332) 5 2 811 285

keyboard.c (411) 2 1 1,256 9

af_inet.c (48) 1 1 1,273 79

eata_pio.c (183) 3 1 984 8

cdc_acm.c (468) 5 3 849 119

ip6_output.c (198)

0 0 1,110 45

mptbase.c (777) 2 1 6,158 8251

aty128fb.c (98)

2 1 2,466 3671Performed on a Linux 2.6 box with Pentium4 3.2GHz, 4GB RAM


Jaeho Shin 12

Performance 2/2

GNU SoftwareAlarm

sReal Errors LOC Time (sec)

tar-1.13 (2,630)

66 1 20,258 577

bison-1.875 (5,164)

50 0 15,907 809

sed-4.0.8 (461) 29 0 6,053 1154

gzip-1.2.4a (799) 17 0 7,327 794

grep-2.5.1 (187) 2 0 9,297 604Commercial SoftwareAlarm

sReal Errors LOC Time (min)

A 18 9 280,379 8

B 196 563,584,66

4789

C 78 15 119,211 82

D 435 7 806,829 112

E 197 112 517,314 8


Jaeho Shin 13

Statistical Post Analysis

1. We collect Samples of true and false alarm Symptoms of each alarm

2. From them, compute trueness of alarms i.e. probability being true given its symptoms

3. With trueness we can Sift out false alarms Report truer alarms first


Jaeho Shin 14

Symptoms Syntactic symptoms

- AfterLoop, AfterBranch, AfterReturn, InNestedLoopBody, InNestedBranchBody

+ InLoopCond, InBranchCond, InFunParam, InNestedFunParam, InRightOfAnd

Semantic symptoms- JoinN, NotNarrowed, ComplexData, InCyclicCallChain+ Prunning, PassedValue, ConstantVariable, ConstantIndex, Consta

ntArrayConstantIndex Result symptoms

- TopIndex, HalfInfiniteIndex+ FiniteOffsetFiniteArray, FiniteIndex

Common-sense + shallow inside info

f

g

h

[9, 10][9, 10][9, 10][9, 10]


Jaeho Shin 15

Bayesian Analysis For each alarm, we compute its conditional

probability being true given its symptoms

Numbers from “learning samples” Estimated using Monte-Carlo method

We assume symptoms occur independently (naïve Bayesian filtering)

We assume symptoms occur independently (naïve Bayesian filtering)


Jaeho Shin 16

Sifting Out Threshold

User’s knob: his/her risk ratio (Rs/Rr)

Minimize risk expectation Risk expectation of an alarm with probability p when

Silencing = Rs x p Reporting = Rr x (1 – p)

We silence if Rs x p < Rr x (1 – p) Hence, sift out when p < Rr / (Rr + Rs)

Risk oftrue

errorsfalse

alarms

silencing

Rs 0

reporting

0 Rr

= 1 / (1 + Rs/Rr)


Jaeho Shin 17

Experiments

With alarms from Parts of the Linux kernel Programs in algorithm text-books

Learning and testing 50%/50% randomly chosen

15 times repeated


Jaeho Shin 18

Sifting Out Alarms Rs = 3 x Rr threshold = 0.25 74.84% of false alarms filtered out :-) 31.40% of true alarms were also swept out :-(


Jaeho Shin 19

Ranking Alarms Show user “truer” alarms first 15.17% of false alarms are mixed up

until the user sees 50% of the true alarms


Jaeho Shin 20

Binary Logistic Regression

Trueness of an alarm given its binary symptom vector

Generalized linear model Coefficients from learning set For example,


Jaeho Shin 21

Bayesian vs. Logistic Regression 1/2

With threshold 0.25, Bayesian: 74.84% of false, 31.40% of true Logistic Regression: 90.05% of false, 20.85% of true

alarms can be sifted out


Jaeho Shin 22

Bayesian vs. Logistic Regression 2/2

Until user sees 50% of true alarms Bayesian: 15.17% Logistic Regression: 4.10%

of false alarms were mixed upConjecture:Logistic regression model respects symptom dependency?

Conjecture:Logistic regression model respects symptom dependency?


Jaeho Shin 23

Related Work

Buffer overrun detection ARCHER [Xie, Chou & Engler 2003] SPLINT [Zitser, Lippmann & Leek 2004] CSSV [Dor, Rodeh & Sagiv 2003] ASTRÉE [Cousot et al. 2005, 2003]

Statistical approach Z-ranking [Kremenek & Engler 2003] Error Correlation [Kremenek et al. 2004]

unsound

require

annotation

domain-aware


Jaeho Shin 24

Conclusion

Our “sound” static analyzer,Airac is realistic

False alarms are inevitablein domain-unaware situation

Statistical approaches helped viable approach to handle false alarms natural symptoms seem to work orthogonal to other static analysis

techniques generic, depends on learning set


Jaeho Shin 25

Thank you

Questions?

Demo available at http://ropas.snu.ac.kr/airac

taming false alarms from a domain-unaware c analyzer by a statistical post analysis

Documents

jaeho shin

statistical post analysissas

domainunaware c analyzerby

rankingtaming false

inevitable false alarms

c buffer

rank alarms

ansi c