msr2011 zaman
TRANSCRIPT
1
Security versus Performance Bugs:
A Case Study on
Shahed Zaman, Bram Adams, Ahmed E. HassanSoftware Analysis and Intelligence Lab (SAIL), Queen’s University
2
Costly
Bugs have a high impact on companies
Affect reputation
482 bugs/weekFirefox
3
Most research treats all bugs equally Does this make sense?
4
Performance Security Other bugs
5
Our Study Dimensions
Are security bugs fixed by more experienced
developers?
Are security fixes more complex?
Time People Fix
Are security bugs fixed faster?
6
Study Overview
Bugzilla
CVS
Bug Type And Bug Fix
Identification
Performance
Security
Other Bugs
Metric Extraction
Analysis
7
https://bugzilla.mozilla.org/show_bug.cgi?id=640339
Identification of Security Bugs
8
Identification of Performance Bugs‘slow’, ‘perf’,
‘hang’
Our heuristics have:100 ± 5% precision
80 ± 5% recall
9
Reported Bugs in Firefox
Timeframe of our study
10
Our Study Dimensions
Are security bugs fixed by more experienced
developers?
Are security fixes more complex?
Time People Fix
Are security bugs fixed faster?
11
The lifetime of a Bug
FIXEDNEW ASSIGNED CLOSED
TRIAGE
12
Security bugs are triaged faster
Log(1 + triage time)
46629
179870
Ratio
of B
ugs
X 3.8
13
The lifetime of a Bug
FIXEDNEW ASSIGNED CLOSED
FIXING
TRIAGED
FASTER
14
Security Bugs are fixed faster
Log(1 + time between assignment and fix)
Ratio
of B
ugs
15
Rework in the lifetime of a Bug
REOPENED
FIXEDNEW ASSIGNED CLOSEDFIXED
FASTER
TOSSING
TRIAGED
FASTER
16
Security Bugs: tossed & re-opened more often
# of times bug tossing
tossed more !
Ratio
of B
ugs
# of times bug reopened
Ratio
of B
ugs
reopened more !
17
Our Study Dimensions
Are security bugs fixed by more experienced
developers?
Are security fixes more complex?
Time People Fix
Are security bugs fixed faster?
YES!
18
Security bugs are fixed by more experienced developers
Experience in # of days
Ratio
of B
ugs
More experienced
19
Our Study Dimensions
Are security bugs fixed by more experienced
developers?
Are security fixes more complex?
Time People Fix
Are security bugs fixed faster?
YES!YES!
20
Entropy as a measure of Complexity
V W X Y Z0
2
4
6
Fix 2
File
# of
cha
nged
line
sA B C D E
0
2
4
6
Fix 1
File
# of
cha
nged
line
s
More Complex
21
Security fixes are more complex
entropy
Ratio
of B
ugs
22
Our Study Dimensions
Are security bugs fixed by more experienced
developers?
Are security fixes more complex?
Time People Fix
Are security bugs fixed faster?
YES!YES! YES!
23
Security Perf. Security Perf.Fix time +Triage time + ? ?# of reopening +# of tossing +# of developer assigned + = =
Experience +# of files changed + = =Entropy +
more(+) no difference (=) studying (?)
Chrome
+
+
+
+
+
24
Security Perf. Security Perf.Fix time +Triage time + ? ?# of reopening +# of tossing +# of developer assigned + = =
Experience +# of files changed + = =Entropy +
more(+) no difference (=) studying (?)
Chrome
+
+
+
+
+
25
Threats to Validity• Focused on one domain• Use of heuristics in bug type identification• Bug disclosure policies
Non-disclosed security bugs
26YES!