defect reporting and triageweb.eecs.umich.edu/~weimerw/481/lectures/se-10-bugreport.pdf · how...
TRANSCRIPT
Defect ReportingDefect Reportingand Triageand Triage
2
The Story So Far …● Quality assurance is critical to software
engineering. ● Static (code review, inspection, dataflow
analysis) and dynamic (testing, instrumentation) approaches are common
● What happens to all of the bugs those find?
3
One-Slide Summary● A software defect report includes information
and communications related to addressing a software issue. ● Defect reports have many components.
● Defect reports are subject to triage based on severity and priority information.
● Defect reports have a lifecycle that is complicated and non-linear with multiple possible resolutions.
4
Is This Really A Problem?
“Every day, almost 300 bugs appear that need triaging. This is far too much for only the Mozilla programmers to handle.”
- Mozilla Developer
5
Is This Really A Problem?
“Every day, almost 300 bugs appear that need triaging. This is far too much for only the Mozilla programmers to handle.”
- Mozilla Developer, 2005
6
Plus ça change …
7
“Just Yesterday”100,000/year = 270/day: same rate 15 years in!
8
Terminology (1/2)● The software maintenance lexicon is fraught with
ambiguity
● Terms adapted from “standard” engineering, etc.
● A fault is an exceptional situation at run time
● In EE: “short circuit”, “stuck-at fault”
● In CS: “trap”, “exception”
● A defect is any characteristic of a product which hinders its usability for its intended purpose
● IRL: “design defect”, “manufacturing defect”
● In CS: a bug is a static defect in the source code
9
Terminology (2/2)
● A bug report provides information about a defect ● Created by testers, users, tools, etc. ● Often contains multiple types of information● Often tracked in a database
● A feature request is a potential change to the intended purpose (requirements) of software● In CS: an issue is either a bug report or a feature
request (cf. “issue tracking system”)
● Not used here: “mistake”, “error”, etc.
10
These Terms Are Somewhat Subjective
11
Defect Report Lifecycle
● The defect report lifecycle consists of a number of possible stages and actions, including reporting, confirmation, triage, assignment, resolution, and verification. ● Not every defect report follows the same path● The overall process is not linear
● There are multiple entry points, some cycles, and multiple exit points (and some never leave …)
● The status of a defect report tracks its position in the lifecycle (“new”, “resolved”, etc.)
12
Report Lifecycle
● Bugzilla is a widely-used open-source issue tracker
● GitHub's built-in issue tracker is similar (less structured)
13
Bug is Reported
● New bug reports enter the system
14
Bug Report Sources
● Internal● Developers● QA / Testers● Reports are usually detailed, sophisticated
● External● Beta Testers● End Users● Reports are usually more general
15
16
Do End Users Submit Bug Reports?
17
End-User Bug Reports
● Modern view: cannot count on end users to describe bugs in a helpful manner● The larger your user
base is or the more of a “margin” business model you have, the truer this becomes
● Instead: these are aggregated
18
Bug Reporting: GitHub
19
20
The Anatomy of a Bug Report
● What should be in a bug report? What is?
21
Defect Report ComponentsTitle
Status
Assignee
Product
22
Defect Report Attachments
● Screenshots● Videos● Stack Traces● Data Files
● Note: rarely present● Note: may come
from multiple sources
23
“Ideal” Defect Report Comment
24
Defect Reports: Conversations
● Defect reports are not static
● Instead, they are updated over time● Request more info● Assign to a dev● Discuss solutions
● The report is a log of all relevant activity
25
Trivia: Entymology
● This group of insects (order Coleoptera) have hardened wing-cases. They make up 40% of all described insects and 25% of all known animal life-forms. They often feed on plants and fungi, break down animal and plant debris, and eat other invertebrates.
26
Trivia: Music
● This 1967 Beatles effort was their best-selling album in the UK (17x platinum). In addition to the eponymous fictional-group lead track, it includes With a Little Help from My Friends, Lucy in the Sky with Diamonds, and When I'm Sixty-Four. “On release, the album was lauded by the vast majority of critics for its innovations in music production, songwriting and graphic design, for bridging a cultural divide between popular music and legitimate art, and for providing a musical representation of its generation and the contemporary
counterculture.”
27
Trivia: Movies
● This giant, lumbering paranormal monster from the Ghostbusters franchise appears when Gozer tells the heroes that it will take the form of the next thing they think of. Ray tries to think of “the most harmless thing … that could never possibly destroy us.”
28
Psychology: Delayed Gratification
● A child is offered a choice between one reward now or two rewards 15 minutes later.
● Over 600 children took part.● Some would “cover their eyes with their hands
or turn around so that they can't see the tray, others start kicking the desk, or tug on their pigtails, or stroke the marshmallow as if it were a tiny stuffed animal,” while others would simply eat the marshmallow as soon as the researchers left.
29
Psychology: Delayed Gratification
● Results:● A minority choose the single reward immediately● A majority attempted to wait the 15 minutes
● One-third of those who attempted succeeded● Age was a major correlated factor● Trust/belief in reward also a major factor
● This work is well-known because of the associated follow-up studies and correlations
30
Psychology: Delayed Gratification
● The ability to delay gratification also correlates with higher SAT scores
● Brain imaging study of a sample from the original Stanford participants when they reached mid-life showed key differences between those with high delay times and those with low delay times in two areas: the prefrontal cortex (more active in high delayers) and the ventral striatum, (more active in low delayers) when they were trying to control their responses to temptations
31
Psychology: Delayed Gratification
● Also correlates with educational attainment, body-mass index, cognitive and academic competence, and ability to cope with frustration and stress in adolescence
● [ Mischel, Walter; Ebbesen, Ebbe B. 1970. “Attention in delay of gratification”. Journal of Personality and Social Psychology. 16 (2): 329–337 ]
● Implications for SE: “quick and dirty” fix or desire to “just start coding” vs. planning and using an SE process
32
Bug Triage
● Which bugs should we address first?
33
Triage
● Triage is the assignment of degrees of urgency to wounds or illnesses to decide the order of treatment of a large number of patients or casualties
● There are always more defect reports than resources available to address them
● Cost-benefit analysis● How expensive is it to fix this bug?● How expensive is it to not fix this bug?
34
Which Bugs Should We Fix?
● Common Myth:
35
Severity
● Severity is the degree of impact that a defect has on the development or operation of a component or system● “cost of not fixing it” ● Bugzilla severity labels:
36
Priority
● Defect Priority (Bug Priority) indicates the importance or urgency of fixing a defect.
● Phabricator examples:● Needs Triage - Default option, priority has not yet been
determined
● Unbreak Now! - Something is broken and needs to be fixed immediately, setting anything else aside
● High - Someone is working or planning to work on this task soon
● Normal - Less than High, but someone still plans to work on it
● Low - Less than Normal, but someone still plans to work on it
● Lowest - Nobody plans to work on this task
37
Priority Assignment Example
● Phabricator Agile example:
● High priority for tasks committed for the current sprint, or that need to find an owner who can start working on them soon
● Normal priority for tasks that are not critical for the current sprint or candidates for a next sprint
● Low priority for tasks that we can live without, usually sitting in the backlog, sometimes added to a sprint
● “As a rule of thumb, limit High priority task assignments for a single person to three, five in exceptional times.”
38
Severity vs. Priority
● Severity and Priority are often correlated, but are officially independent● A “Normal” Severity issue could currently be
“Low” Priority if there are many outstanding “Critical” Severity issues, etc.
● Severity and Priority are used together (along with complexity, risk, etc.) to evaluate, prioritize and assign the resolution of reports● Distributed on-line voting (e.g., in open source)● In-person meeting (e.g., commercial)
39
Simple Lie
● Supply + Demand Price→● Severity + Priority Triage→
40
Bug Assignment● Who should
fix this bug?
41
Example
● Severity and Priority discussions
● Assignment discussions
42
Defect Assignment
● An assignment associates a developer with the responsibility of addressing a defect report
● The current state of the art is “manual”● Distributed: developers watch the incoming bug
report queue and claim defects for themselves● Centralized: one or more people in QA watch the
incoming bug report queue and assign reports to a pool of developers
● Usually based who “owns” the implicated code
43
Near Future: Automatic Assignment
44
Near Future: Automatic Assignment
Seven Years Later
Seven Years Later
45
Bug Resolution
● Did we fix it?
46
Defect Resolution
● Now that the defect report has been assigned to a developer, it can be localized, debugged, etc. Those are future lecture topics!
● A defect report resolution status indicates the result of the most recent attempt to address it● Important: resolved need not mean “fixed”
47
Possible Resolutions● Bugzilla resolution options:
● FIXED (give commit #)● INVALID (bug report is invalid)● WONTFIX (we don't ever plan to fix it)● DUPLICATE (link to other bug report #)● WORKSFORME (cannot reproduce, a.k.a. “WFM”)● MOVED (give link: filed with wrong project)● NOTABUG (report describes expected behavior)● NOTOURBUG (is a bug, but not with our software)● INSUFFICIENTDATA (cannot triage/fix w/o more)
48
Possible Resolutions● Bugzilla resolution options:
● FIXED (give commit #)● INVALID (bug report is invalid)● WONTFIX (we don't ever plan to fix it)● DUPLICATE (link to other bug report #)● WORKSFORME (cannot reproduce, a.k.a. “WFM”)● MOVED (give link: filed with wrong project)● NOTABUG (report describes expected behavior)● NOTOURBUG (is a bug, but not with our software)● INSUFFICIENTDATA (cannot triage/fix w/o more)
Thought question:What fraction of
bug reportsend up with each
Resolution?
49
Duplicate, Invalid
[ Jalbert et al. Automated Duplicate Detection for Bug Tracking Systems. DSN 2008. ]
50
Reopen?
● I thought we fixed it!
51
Reopened
● A defect report that was previously resolved (e.g. “FIXED”) may be reopened if later evidence suggests the old resolution is no longer adequate● “We thought this fixed it, but now others are
reporting it.”● “We thought this was out of scope, but now we
really need to address it.”
● Compare: regression testing● Surely this only happens rarely?
52
Many Fixes Are WrongEven On Mature, Critical Software
[Yin et al. How Do Fixes Become Bugs? ESEC/FSE 2011. ]
53
Bug Report Characteristics
54
Large Study of Bug Reports
● 2000 defect reports in Linux, Mozilla, Apache● Memory Bugs: ~15%; Semantic Bugs: ~75%;
Concurrency Bugs: ~10%● Bug Crash: ~20%; Bug Wrong Behavior: ~80%→ →
● Why Crash? Memory Bugs ~55%
● Most common? ~50% of Mozilla bugs are GUI issues
● Whence security bugs? 30% memory bug causes (severe), 70% semantic bug causes
[ Tan et al. Bug Characteristics in Open Source Software. EMSE 2014. ]
55
Questions?● Exam 1 and HW3