leakpoint: pinpointing the causes of memory leaks (icse 2010)

LEAKPOINT: Pinpointing the Causes of Memory Leaks

Georgia Institute of TechnologyJames Clause and Alessandro Orso

Supported in part by NSF and IBM Research

Memory leak classification

void *p = malloc(100);


void *p = malloc(100); M


Lost memory Forgotten memory

M becomes unreachable before being deallocated

M is reachable, but is never accessed or deallocated



Lost memory Forgotten memory

M becomes unreachable before being deallocated

M is reachable, but is never accessed or deallocated


• common

• difficult to manually detect

• high impact

Existing techniques

mtrace

M. Bond and K. McKinley ‘06

R. Hastings and B. Joyce ‘92

M. Hauswirth and T. Chilimbi. ‘04

D.Heine and M.Lam ‘03

D. Heine and M. Lam ‘06

M. Jump and K. McKinley ‘07

leaks

J. Maebe, M. Ronsse, and K. D. Bosschere ‘04

N. Mitchell and G. Sevitsky ‘03

G. Novark, E. D. Berger, and B. G. Zorn ‘09

M. Orlovich and R. Rugina ‘06

F. Qin, S. Lu, and Y. Zhou ‘05

MemCheck

Y. Xie and A. Aiken ‘05

G. Xu and A. Rountev ‘08

S. Cherem, L. Princehouse, and R. Rugina ‘06

W. DePauw and G. Sevitsky ’99purify

Publications Tools

addhash(char hname[]) {35. int i;36. HASHPTR hptr;37. unsigned int hsum = 0;

38. for(i = 0 ; i < strlen(hname) ; i++) {39. sum += (unsigned int) hname[i];40. }41. hsum %= 3001;42. if((hptr = hashtab[hsum]) == (HASHPTR) NULL) {43. hptr = hashtab[hsum] = (HASHPTR) malloc(sizeof(HASHBOX));44. hptr->hnext = (HASHPTR) NULL;45. hptr->hnum = ++netctr;46. hptr->hname = (char *) malloc((strlen(hname) + 1) * ! ! ! ! ! ! ! ! ! ! sizeof(char));47. sprintf(hptr->hname , "%s" , hname);48. return(1);49. } else {! ... 67. } }

Detecting leaks is easy



Detecting leaks is easy; fixing them is not

Overview

Overview

Leak locations are close to whereleaks should be fixed.

Overview

1 Taintingpointers


Overview

1 Taintingpointers

2 Propagatingtaint marks


Overview

1 Taintingpointers


3 Identifyingwhen leaksoccur


46. hptr->hname = (char *) malloc((strlen(hname) + 1) * ! ! ! ! ! ! ! ! ! ! sizeof(char));

delHtab() {15. int i;16. HASHPTR hptr , zapptr;17. for(i = 0; i < 3001; i++) {18. hptr = hashtab[i];19. if(hptr != (HASHPTR) NULL) {20. zapptr = hptr ;21. while(hptr->hnext != (HASHPTR) NULL) {22.! ! hptr = hptr->hnext;23.! ! free(zapptr);24.! ! zapptr = hptr ;25.! ! }

26.! ! free(hptr);27.! }28. }!29. free(hashtab);30. return; }



Detecting leaks is easy; fixing them is too




Detecting leaks is easy; fixing them is too



free(hptr->hname);

Outline

• Our technique

• Tainting pointers

• Tracking pointers

• Checking for leaks

• Implementation

• Evaluation

• Conclusions and future work

1. Tainting pointersAssign a taint mark to pointers returned from

allocation functions (e.g., malloc)

1. Tainting pointers

Last use location

Allocation location

Allocation sizeDeallocated indicatorPointer count

Assign a taint mark to pointers returned from allocation functions (e.g., malloc)

Metadata

1. Tainting pointers

Last use location

Allocation location

Allocation sizeDeallocated indicatorPointer count

Assign a taint mark to pointers returned from allocation functions (e.g., malloc)

current location

current location

false

size of the memory area

1

Metadata Initialized to

2. Propagating taint marks


1. Track the flow of pointers throughout the execution


1. Track the flow of pointers throughout the execution

2. Update taint marks’ mutable metadata

Tracking pointers

assignment

addition subtraction

and

multiplicationdivision

modulusor, xor,

shift, notcomparison

p2 = p1 ➔ p2

p2 = p1 ➔ p2

p2 = p2 ± 1 ➔ p2

p3 = p2 ± p1 ➔ p3

p2 = p2 & 0xffff ➔ p2

not tainted

Tracking pointers

assignment


and


modulusor, xor,


p2 = p1 ➔ p2

p2 = p1 ➔ p2

p2 = p2 ± 1 ➔ p2

p3 = p2 ± p1 ➔ p3

p2 = p2 & 0xffff ➔ p2

not tainted

Tracking pointers

Based on domain knowledge and expertise

assignment


and


modulusor, xor,


Update metadata (1)

Pointer Counts

Update metadata (1)

Pointer Counts

• Assignment: increment the count of the pointer that is copied, decrement the count of the pointer that is overwritten

Update metadata (1)

Pointer Counts


Update metadata (1)

ptr3 = ptr1 ➔ ptr3 , ptr1 1 2 2

ptr1 = NULL ➔ ptr1 , ptr32 1

Pointer Counts


• Function return: decrement the count of pointers stored in local variables

Update metadata (1)

ptr3 = ptr1 ➔ ptr3 , ptr1 1 2 2


Pointer Counts


• Function return: decrement the count of pointers stored in local variables

• Memory deallocation: decrement the count of pointers reachable from the deallocated memory

Update metadata (1)

ptr3 = ptr1 ➔ ptr3 , ptr1 1 2 2


Update metadata (2)

Deallocation indicator

Update metadata (2)


• Set to true when a pointer is passed to a deallocation function (e.g., free)

Update metadata (2)



Last use location

Update metadata (2)



Last use location

• Set to the current location whenever a pointer is

- propagated- passed as a function argument- returned from a function- used to access memory

Update metadata (2)

3. Identifying when leaks occur

3. Identifying when leaks occurLost memory Forgotten memory

If a taint mark’s pointer count is zero and it’s deallocated

indicator is false

If, at the end of execution, a taint mark’s deallocated

indicator is false



indicator is false


indicator is false

(Checks are recursive)



indicator is false


indicator is false

Generate a leak report: • allocation location, allocation size, and last use location




indicator is false


indicator is false

Generate a leak report: • allocation location, allocation size, and last use location

Merge leak reports: • combine reports with identical allocation and last use

locations, add allocation sizes


Prototype tool

Implemented using Valgrind

Prototype tool


30–100x overheads

Prototype tool

16 bytes of memory allocated: at malloc by addhash (hash.c:50) by parser (parser.c:210) by readcell (parser.c:34) by main (main.c:98) was leaked: at free by delHtab (hash.c:28) by grdcell(grdcell.c:354) by main (main.c:227)


Prototype tool

16 bytes of memory allocated: at malloc by addhash (hash.c:50) by parser (parser.c:210) by readcell (parser.c:34) by main (main.c:98) was leaked: at free by delHtab (hash.c:28) by grdcell(grdcell.c:354) by main (main.c:227)


Can be used to prioritize debugging effort

Evaluation

How does Leakpoint’s ability to detect memory leaks compare

to existing tools?

How effective is Leakpoint at guiding developers to the

locations where memory leaks may be fixed?

RQ1: Comparison with existing tools

Subjects

164.gzip 4 1 4 4175.vpr 47 0 47 47176.gcc 1121 406 (1415) 1121 1121181.mcf 0 0 0 0

186.crafty 37 0 37 37197.parser 2 0 2 2

252.eon 380 380 380 380253.perlbmk 3481 0 (2) 3481 536

254.gap 2 0 (2) 2 2255.vortex 15 1 15 15256.bzip2 10 1 10 10300.twolf 1403 68 (3) 1403 1403


Subjects

164.gzip 4 1 4 4175.vpr 47 0 47 47176.gcc 1121 406 (1415) 1121 1121181.mcf 0 0 0 0

186.crafty 37 0 37 37197.parser 2 0 2 2

252.eon 380 380 380 380253.perlbmk 3481 0 (2) 3481 536



Leakpoint

Subjects

164.gzip 4 1 4 4175.vpr 47 0 47 47176.gcc 1121 406 (1415) 1121 1121181.mcf 0 0 0 0

186.crafty 37 0 37 37197.parser 2 0 2 2

252.eon 380 380 380 380253.perlbmk 3481 0 (2) 3481 536



omegaLeakpoint

Subjects

164.gzip 4 1 4 4175.vpr 47 0 47 47176.gcc 1121 406 (1415) 1121 1121181.mcf 0 0 0 0

186.crafty 37 0 37 37197.parser 2 0 2 2

252.eon 380 380 380 380253.perlbmk 3481 0 (2) 3481 536



omega MemCheckLeakpoint

Subjects

164.gzip 4 1 4 4175.vpr 47 0 47 47176.gcc 1121 406 (1415) 1121 1121181.mcf 0 0 0 0

186.crafty 37 0 37 37197.parser 2 0 2 2

252.eon 380 380 380 380253.perlbmk 3481 0 (2) 3481 536



mtraceomega MemCheckLeakpoint

Leak detectionLeak identificationSubjects

164.gzip 4 1 4 4175.vpr 47 0 47 47176.gcc 1121 406 (1415) 1121 1121181.mcf 0 0 0 0

186.crafty 37 0 37 37197.parser 2 0 2 2

252.eon 380 380 380 380253.perlbmk 3481 0 (2) 3481 536




Subjects

164.gzip 4 1 4 4175.vpr 47 0 47 47176.gcc 1121 406 (1415) 1121 1121181.mcf 0 0 0 0

186.crafty 37 0 37 37197.parser 2 0 2 2

252.eon 380 380 380 380253.perlbmk 3481 0 (2) 3481 536




# Detected memory leaks (# false positives)Subjects

164.gzip 4 1 4 4175.vpr 47 0 47 47176.gcc 1121 406 (1415) 1121 1121181.mcf 0 0 0 0

186.crafty 37 0 37 37197.parser 2 0 2 2

252.eon 380 380 380 380253.perlbmk 3481 0 (2) 3481 536




# Detected memory leaks (# false positives)Subjects

164.gzip 4 1 4 4175.vpr 47 0 47 47176.gcc 1121 406 (1415) 1121 1121181.mcf 0 0 0 0

186.crafty 37 0 37 37197.parser 2 0 2 2

252.eon 380 380 380 380253.perlbmk 3481 0 (2) 3481 536




Leakpoint is at least as effective as existing tools at detecting memory leaks

RQ2: Effectiveness at guiding developers

Compare the leak locations identified by Leakpoint with the locations where the leaks were fixed by

the original application developers.




Transmission




Transmission4 memory leaks total

static void processCompletedTasks(tr_web *web) { ... task->done_func(web->session, ..., task->done_func_user_data); ... evbuffer_free(task->response); tr_free(task->url); tr_free(task); ...}

Transmissionstatic void invokeRequest(void * vreq) { ... hash = tr_new0(uint8_t, SHA_DIGEST_LENGTH); memcpy(hash, req->torrent_hash, SHA_DIGEST_LENGTH); tr_webRun(req->session, req->url, req->done_func, hash); ...}



static void onStoppedResponse(tr_session *session, ..., void *torrent_hash) { dbgmsg(NULL, "got a response ... message");

onReqDone(session);}

// tr_free(torrent_hash);



static void onStoppedResponse(tr_session *session, ..., void *torrent_hash) { dbgmsg(NULL, "got a response ... message");

onReqDone(session);}

// tr_free(torrent_hash);

Distance: 6 statements

URIHANDLER_FUNC(mod_rewrite_uri_handler) { ...

hctx = handler_ctx_init();

con->plugin_ctx[p->id] = hctx;

...}

Lighttpd 1




...}

Lighttpd 1

// if(con->plugin_ctx[p->id] == NULL) {

// } // else { // hctx = con->plugin_ctx[p->id]; // }




...}

Lighttpd 1

// if(con->plugin_ctx[p->id] == NULL) {

// } // else { // hctx = con->plugin_ctx[p->id]; // }

Distance: overlapping

int http_request_parse(server *srv, connection *con) { ... if(NULL == (ds = (data_string *)array_get_unused_element( con->request.headers, TYPE_STRING))) { ds = data_string_init(); } ... else if (cmp > 0 && 0 == (cmp = buffer_caseless_compare(CONST_BUF_LEN(ds->key), CONST_STR_LEN("Content-Length")))) { char *err unsigned long int r; size_t j if (con_length_set) { con->http_status = 400; con->keep_alive = 0; if(srv->srvconf.log_request_header_on_error) { log_error_write(srv, __FILE__, __LINE__, "s", "duplicate ..."); log_error_write(srv, __FILE__, __LINE__, "Sb", "request-header:\n", con->request.request); }

return 0; } ...}

Lighttpd 2


return 0; } ...}

Lighttpd 2

// array_insert_unique(con->request.headers, (data_unset *)ds);


return 0; } ...}

Lighttpd 2

// array_insert_unique(con->request.headers, (data_unset *)ds);

Distance: 1 statement

static struct spelling *spelling_base;

static void push_string(char *string) { ... spelling_base = xmalloc(spelling_size * sizeof(struct spelling)); ...}

void finish_init() { ...

constructor_decl = p->decl; ... spelling_base = p->spelling_base; ...}

GCC





GCC

// free(spelling_base);





GCC

// free(spelling_base);

Distance: 10 statements*

Summary

• A new technique for identifying where memory leaks occur

• at least as effective as existing techniques at detecting memory leaks

• helpful in guiding developers to the locations where memory leaks should be fixed

Future work

Future work

Improvedimplementation

Future work

Additionalexperimentation


Future work

Additionalexperimentation

UserStudies


Questions?

1 Taintingpointers


3 Identifyingwhen leaksoccur

leakpoint: pinpointing the causes of memory leaks (icse 2010)

Technology

hashptr null45

hashptr mallocsizeofhashbox44

hashptr hptr37

unsigned int hsum

addhashchar hname

int i36

unsigned int hnamei40

int i16