msr2011 zaman

1

Security versus Performance Bugs:

A Case Study on

Shahed Zaman, Bram Adams, Ahmed E. HassanSoftware Analysis and Intelligence Lab (SAIL), Queen’s University

2

Costly

Bugs have a high impact on companies

Affect reputation

482 bugs/weekFirefox

3

Most research treats all bugs equally Does this make sense?

4

Performance Security Other bugs

5

Our Study Dimensions

Are security bugs fixed by more experienced

developers?

Are security fixes more complex?

Time People Fix

Are security bugs fixed faster?

6

Study Overview

Bugzilla

CVS

Bug Type And Bug Fix

Identification

Performance

Security

Other Bugs

Metric Extraction

Analysis

7

https://bugzilla.mozilla.org/show_bug.cgi?id=640339

Identification of Security Bugs

8

Identification of Performance Bugs‘slow’, ‘perf’,

‘hang’

Our heuristics have:100 ± 5% precision

80 ± 5% recall

9

Reported Bugs in Firefox

Timeframe of our study

10

Our Study Dimensions

Are security bugs fixed by more experienced

developers?

Are security fixes more complex?

Time People Fix

Are security bugs fixed faster?

11

The lifetime of a Bug

FIXEDNEW ASSIGNED CLOSED

TRIAGE

12

Security bugs are triaged faster

Log(1 + triage time)

46629

179870

Ratio

of B

ugs

X 3.8

13

The lifetime of a Bug

FIXEDNEW ASSIGNED CLOSED

FIXING

TRIAGED

FASTER

14

Security Bugs are fixed faster

Log(1 + time between assignment and fix)

Ratio

of B

ugs

15

Rework in the lifetime of a Bug

REOPENED

FIXEDNEW ASSIGNED CLOSEDFIXED

FASTER

TOSSING

TRIAGED

FASTER

16

Security Bugs: tossed & re-opened more often

# of times bug tossing

tossed more !

Ratio

of B

ugs

# of times bug reopened

Ratio

of B

ugs

reopened more !

17

Our Study Dimensions

Are security bugs fixed by more experienced

developers?

Are security fixes more complex?

Time People Fix

Are security bugs fixed faster?

YES!

18

Security bugs are fixed by more experienced developers

Experience in # of days

Ratio

of B

ugs

More experienced

19

Our Study Dimensions

Are security bugs fixed by more experienced

developers?

Are security fixes more complex?

Time People Fix

Are security bugs fixed faster?

YES!YES!

20

Entropy as a measure of Complexity

V W X Y Z0

2

4

6

Fix 2

File

# of

cha

nged

line

sA B C D E

0

2

4

6

Fix 1

File

# of

cha

nged

line

s

More Complex

21

Security fixes are more complex

entropy

Ratio

of B

ugs

22

Our Study Dimensions

Are security bugs fixed by more experienced

developers?

Are security fixes more complex?

Time People Fix

Are security bugs fixed faster?

YES!YES! YES!

23

Security Perf. Security Perf.Fix time +Triage time + ? ?# of reopening +# of tossing +# of developer assigned + = =

Experience +# of files changed + = =Entropy +

more(+) no difference (=) studying (?)

Chrome

+

+

+

+

+

24

Security Perf. Security Perf.Fix time +Triage time + ? ?# of reopening +# of tossing +# of developer assigned + = =

Experience +# of files changed + = =Entropy +

more(+) no difference (=) studying (?)

Chrome

+

+

+

+

+

25

Threats to Validity• Focused on one domain• Use of heuristics in bug type identification• Bug disclosure policies

Non-disclosed security bugs

26YES!