2017 IEEE 28th International Symposium on Software Reliability Engineering
Experience Report: Study of Vulnerabilities
of Enterprise Operating Systems
Anatoliy Gorbenko Alexander Romanovsky
School of Computing, Creative School of Computing Science,
Technologies & Engineering, Newcastle University,
Leeds Beckett University, Newcastle-upon-Tyne, UK
Leeds, UK Alexander.Romanovsky
A.Gorbenko@leedsbeckett.ac.uk @ncl.ac.uk
Abstract—This experience report analyses security problems of
modern computer systems caused by vulnerabilities in their
operating systems. An aggregated vulnerability database has
been developed by joining vulnerability records from two
publicly available vulnerability databases: the Common
Vulnerabilities and Exposures system (CVE) and the National
Vulnerabilities database (NVD). The aggregated data allow us
to investigate the stages of the vulnerability life cycle,
vulnerability disclosure and the elimination statistics for
different operating systems. The specific technical areas the
paper covers are the quantitative assessment of vulnerabilities
discovered and fixed in operating systems, the estimation of
time that vendors spend on patch issuing, and the analysis of
the vulnerability criticality and identification of vulnerabilities
common for different operating systems.
Keywords: security, vulnerability, operating systems,
vulnerability databases, days-of-risk, forever-day vulnerabilities,
vulnerability life cycle, vulnerability statistics
I. INTRODUCTION
Security of information and communication systems has
become one of the most crucial concerns for both system
developers and users. Recent security incidents like those
happened to Hollywood Presbyterian Medical Center [1] or
San Francisco Municipal Transportation Agency [2] show
how vulnerable to attacks our modern society is. They may
cost millions of US dollars and even affect human lives.
One of the main reasons of successful attacks, malicious
intrusions and virus infections are software vulnerabilities in
computer systems, communication equipment, smartphones
and other intellectual devices.
Generally speaking, vulnerability is a weakness which
allows an intruder to undermine system’s information
assurance. MITRE Corporation defines vulnerability as a
software fault that can be directly used by a hacker to gain
access to a system or network [3]. Exploiting vulnerability
allows attackers to either execute commands as normal users,
or access data violating the specified access restrictions or
cause denial of service attack and terminate system services.
Software vulnerabilities are mainly caused by errors and
weaknesses in software design and implementation. For
instance, in 2016 Microsoft Inc. issued 561 updates for its
family of operating systems [4]. 311 of them were security
2332-6549/17 $31.00 © 2017 IEEE
DOI 10.1109/ISSRE.2017.20
Olga Tarasyuk Oleksandr Biloborodov
Department of Computer Plarium Ukraine LLC,
Systems and Networks, Kharkiv, Ukraine
National Aerospace University, Alexandr.Bright@mail.ru
Kharkiv, Ukraine
O.Tarasyuk@csn.khai.edu
updates eliminating software vulnerabilities, among which
114 updates were ranked as critical. During the first 4 months
of 2017 Microsoft has issued 63 security updates out of 99.
Vulnerabilities can be discovered in both application
software and operating systems. For instance, CVE-2016-
7256 vulnerability refers to a weakness in the Windows
family of operating systems when the font library improperly
handles specially crafted embedded OpenType fonts [5, 6, 7].
An attacker can successfully exploit this vulnerability through
web-based or file sharing attack scenarios. As a result, a full
control upon the affected system can be taken allowing
hackers to install programs, view, change or delete data,
create new user accounts with administrative rights, etc.
Another example is the CVE-2014-0160 vulnerability [8] in
the OpenSSL cryptography library that allows remote
attackers to obtain sensitive information from process
memory and even to compromise server secret key via crafted
packets that trigger a buffer over-read. It has affected half a
million widely trusted websites and services including Yahoo,
Amazon Web Services, GitHub, Wikipedia, etc. This
vulnerability existed since December, 2012 and was disclosed
only in April, 2014.
There is no doubt that vulnerabilities in operating
systems are the most critical security threats as they allow to
violate application-oriented access control [9]. Besides, their
exploitation can compromise all processes and services
running in the operating system and allow attackers to gain
access to all data stored on the vulnerable computer. They
represent threats to system security and dependability that
are additional to faults, errors and failures traditionally dealt
with by the dependability community.
In the paper we examine vulnerabilities of popular
enterprise operating systems, investigate statistics of
vulnerability disclosure and elimination and analyze their
criticality. Our results have been obtained by aggregating
statistics provided by the two most reputable vulnerability
databases CVE (www.cve.mitre.org) and NVD
(web.nvd.nist.gov). In contrast to other works investigating
software vulnerabilities [10, 11, 12] this paper focuses on
examining how vulnerability disclosure rates have been
changing over the last years for particular operating systems
and how much time OS vendors spend on issuing patches to
fix that security flaws and how many yet unfixed
205
vulnerabilities can exist in each OS simultaneously. In
addition to this we expand the research results reported in
the earlier works [13, 14, 15, 16, 17] by investigating the
trends in vulnerability severity and discussing the most
common vulnerability types.
Forewarned is forearmed! We believe our study is one of
the first which quantitatively demonstrates that (i) a period of
time between a vulnerability is disclosed and the patch
becomes available can be up to several months long; (ii) a
number of forever-day vulnerabilities can reach several
dozen; (iii) each OS has only few if any known vulnerability-
free days during its lifetime. The reported statistics will also
help system administrators facing a challenge of choosing an
operating system for secure hosting of corporate/web
services to make a grounded decision.
The rest of the paper is organized as follows. In the next
section we briefly describe the research methodology used.
Section III discusses the vulnerability life cycle and a notion
of days-of-risk metric defining how fast software vendors
react on vulnerabilities discovered in their products.
In section IV we investigate various aspects of OS
vulnerability and present vulnerabilities discovery and
elimination statistics, discuss vulnerability severity and
analyse vulnerabilities discovered in more than one operating
system. Finally, some practical lessons learnt from our work
are summarized in section V.
II. VULNERABILITY DATABASES AND RESEARCH
METHODOLOGY
There are a lot of institutions focusing their activity on
vulnerability discovery and elimination. They include,
undoubtedly, software vendors as well as a number of
governmental and independent international organizations,
commercial enterprises and even individuals. Most of these
institutions provide publically available vulnerability
datasets. The most known and trusted are:
• CVE – the Common Vulnerabilities and Exposures
system provided by not-for-profit MITRE Inc.
(cve.mitre.org). MITRE maintains a list of known
vulnerabilities and performs their enumeration by
assigning CVE-IDs which are used by many others
vulnerability databases to synchronize with CVE and
enable data exchange between security databases
and products. In 2016 MITRE has assigned more
than 9900 vulnerability identifiers.
• NVD is the National Vulnerability Database
provided by U.S. National Institute of Standards and
Technology (web.nvd.nist.gov). NVD offers a
dataset of software security vulnerabilities which is
based upon and synchronized with CVE. It classifies
vulnerability severity and type and also specifies
vulnerable software and provides additional meta-
data using the Common Vulnerability Scoring
System (CVSS), the Common Weakness
Enumeration Specification (CWE) and the Common
Platform Enumeration Dictionary (CPE). It has
reported almost 6000 vulnerabilities disclosed in
2016 which is 16.5 vulnerabilities per day in
206
average. About half of them have been observed in
operating systems.
• VNDB is the Vulnerability Notes Database provided
by CERT (www.kb.cert.org/vuls/). Most of VNDB
entries are covered by CVE and NVD. Though, it
was reported that the CERT VNDB vulnerability
disclosures appear on its website 24–72 hours before
they appear in CVE or NVD [18].
• VulnDB is the Risk Based Security’s vulnerability
database (www.riskbasedsecurity.com/vulndb/).
VulnDB tracks vulnerabilities in third-party libraries
and claims to provide over 47,000 vulnerabilities
that are not found in CVE or NVD. Though, its
commercial offering prevents VulnDB from been
widely used by researches.
• SecurityTracker is another vulnerability dataset
commercially available at securitytracker.com. It
almost entirely covers vulnerability entries that have
CVE IDs.
Besides, many software product vendors often provide
information about vulnerabilities in their products as through
the security bulletins (e.g. https://technet.microsoft.com/en-
us/security/bulletins.aspx). At the same time, OSVDB (Open
Source Vulnerability Database) and FVDB (Frei's
Vulnerability Database) that recently have been actively used
by many researchers seem to be no longer available.
Both, CVE and NVD offer access to vulnerability data
sets through the simple search interface available on their web
sites or by distributing XML data feeds. Unfortunately, they
do not support SQL querying making difficult a direct use of
the CVE and NVD repositories for complex analytics. In our
study we have merged together XML data files provided by
CVE and NVD and inserted the joint data set into MySQL
database. We use CVE-ID as a primary key to uniquely
identify each vulnerability. CPE identifiers provided by NVD
are used to assign a particular vulnerability to a certain
product out of the three main groups: (i) operating systems,
(ii) application software and (iii) hardware components (e.g.
routers, graphical cards, embedded devices, etc.). Besides, we
store two date items associated with the same vulnerability:
when it firstly appears in CVE and when its description is
published by NVD.
By now, our joined MySQL database includes more than
100000 vulnerabilities (see Table I). The list of vulnerable
software for 4065 vulnerabilities includes both operating
systems and application software. 641 vulnerabilities have
been discovered simultaneously in operating systems and
hardware components, and 555 vulnerabilities – in
application software and hardware.
Besides, 85 vulnerabilities have been observed in all three
groups. Some of vulnerabilities stored in CVE and NVD are
marked as Reject (i.e. a vulnerability existence was not
confirmed by further investigation), Disputed (i.e. software
vendor does not approve vulnerability) or Deprecated (i.e. a
vulnerability duplicates one of earlier).
All the rejected and deprecated vulnerabilities have been
taken out of further consideration. In our work we
investigated six popular enterprise operating systems (see.
Table II).
 TABLE I. AMOUNT OF VULNERABILITIES version, analyzed in the paper. This can be explained if we
IN THE AGGREGATED SQL DATABASE
assume that a significant piece of code of the most recent
Type of
CPE ID
Amount of Vulnerability attribute 11.3 version has been reused from the 10th version instead
product vulnerabilities Disputed Reject Deprecated on the 11th.
Operating The vulnerabilities of the earlier versions of some of those
cpe:/o: … 14080 25 0 1
system
OSs were investigated by other authors in 2000 [15] and 2006
Application
software
cpe:/a: … 67956 405 0 2 [13, 16, 17]. In this paper we continue and compare the
Hardware cpe:/h: … 2794 9 0 0 vulnerability trends and examine four novel security aspects
Not that can give important insights for OS vendors,
no cpe-id 22242 3 934 0
defined administrators, system security engineers and ordinary users:
Total: 107072 442 934 3 1) quantitative analysis and statistics comparison of
disclosed and fixed vulnerabilities for different operating
There were several reasons for choosing particular OSes systems;
and their versions: popularity, belonging to different types 2) assessment and analyzing number of days-of-risk and
(proprietary and open-source) and families (Windows, forever-day vulnerabilities for each operating system during
Unix/Linux, MacOS), and vendors. In particular, we took their operating cycle;
into consideration a series of reports [19, 20, 21] analysing 3) comparison of vulnerabilities severity and analyzing
OS market share for web- (where Linux-based OSs are types of the most numerous vulnerabilities discovered in
dominating) and on-premises (mainly occupied by different various OSs;
versions of Microsoft Windows Server) servers. 4) discovery of vulnerabilities that are common for two
Our intention was to study vulnerability statistics during or more operating systems.
a considerable period of time to capture general trends and to Thus, one of our intentions was to analyse how their
make sure that our results are well-grounded due to using a security and vulnerability have been changed since that time.
comprehensive dataset (CVE and NVD report statistically Ubuntu Server, Red Hat Enterprise Linux and Novell
insufficient information regarding the most recent versions Linux Enterprise Server are non-monolithic operating
of OSes). This is why the versions of operating systems we systems, thus in our study we also considered vulnerabilities
chose (see Table II) were released in 2012 (all of them are in Linux kernels they use.
still supported by manufacturers 7), and our vulnerability
study covers 5 years preceding the end of 2016. III. SOFTWARE VULNERABILITY LIFECYCLE
Despite the fact that the selected OSs have been now Software vulnerability lifecycle has been discussed in a
replaced by newer versions, our analysis shows that a number of research papers [11, 12, 22]. The authors of [23]
significant number of new vulnerabilities are still disclosed proposed a formal model of the vulnerability lifecycle
in the chosen OSs versions. Unfortunately the majority of defining its major milestones. Most of the researchers and
those vulnerabilities exists in the most recent operating security analysts mark out 5 main events in a typical
systems too. vulnerability lifecycle: (i) vulnerability creation;
TABLE II. OPERATING SYSTEMS UNDER INVESTIGATION
(ii) vulnerability discovery; (iii) vulnerability disclosure;
(iv) patch availability; (v) patch installation.
Release Linux kernel Besides, exploits or computer viruses can be released in a
Operating system
date version wild during vulnerability lifecycle. An exploit is a sequence
Ubuntu Server 12.04 26.04.2012 3.2.x
of commands, a software tool or even a specially generated
Red Hat Enterprise Linux 6 10.11.2010 2.6.32
Novell Linux SUSE data (e.g. an infected file) which automates making use of a
27.02.2012 3.0.x vulnerability and allows even unskilled users to attack
Enterprise Server 11 SP2
Microsoft Windows Server 2012 R2 18.10.2012 - computer systems. Thus, exploit availability is an additional
Apple MacOS Server 10.8 25.06.2012 - event sliding in between events (i) and (v).
Oracle/Sun Solaris 11 09.11.2011 - Time intervals between the above mentioned events in
the vulnerability life cycle have different risks of system
For instance, 100% of Apple MacOS Server’s exposure associated with them.
vulnerabilities reported by the NVD database during 2016 In particular, a special term days-of-risk [22] is used to
were discovered in both 10.8 and 10.11 versions. The last define a period of an increased security risk between the time
one was released on 21.03.2016. The percentages of the when a vulnerability is discovered or publicly disclosed to
vulnerabilities, common for 2012 and 2016 (released on the time when a patch is applied to fix it.
26.09.2016) versions of Microsoft Windows Server, 6.x and Usually the periods of black, gray and white risk are
7.x (released on 10.06.2014, last updated on 01.08.2017) marked out to indicate public awareness of the hazard and to
versions of Red Hat Enterprise Linux, and 12.4 and 16.4 qualify relative risks of exposure (see Fig. 1).
(released on 21.04.2016) versions of Ubuntu Server are equal Risk value depends on vulnerability severity and other
to 85%, 75% and 70% correspondingly. factors. It dramatically increases when an exploit is released
It is also worth noting that Oracle Solaris 11.3 shares in the wild and comes down when software vendor issues a
about 30 percentages of vulnerabilities discovered during patch to fix vulnerability. Application of anti-virus software,
2016 with the Oracle Solaris 10.0 but none with the 11.0 firewalls, intrusion detection systems mitigate risk value.

207
 IV. OPERATING SYSTEMS VULNERABILITY
A. Statistics of vulnerability discovery and elimination
In this section we summarize statistics of vulnerabilities
discovered and disclosed in different operating systems since
the 1st of January, 2012 and until the 31st of December, 2015
(see Table III). A number of vulnerabilities that had been
observed but not fixed by the 1st of January, 2012 are
reported as ‘Starting’.
TABLE III. OPERATING SYSTEMS VULNERABILITY STATISTICS

Windows

Red Hat

MacOS
Ubuntu

Solaris
Novell
Year
Vulnerabilities

Starting number 15 0 46 26 0 13
Disclosed 64 10 31 32 2 47
Fixed 32 5 40 36 2 47

2012
Avg.Sev. 5.21 8.31 5.07 5.13 3.20 4.37
Avg.DoGR 144 132 243 109 94 89
Disclosed 196 59 71 124 60 31
Figure 1. Vulnerability lifecycle and risk of exposure. Fixed 202 51 86 127 59 32

2013
Avg.Sev. 5.04 7.12 5.09 4.94 4.92 4.72
How public vulnerability disclosure (e.g. through CVE or Avg.DoGR 109 131 119 99 113 81
NVD) affects system security is a question of great debates Disclosed 188 64 72 129 40 37
[24, 25]. On the one hand, malicious hackers can use 2014 Fixed 197 38 58 107 40 35
publicly available information to attack affected computer Avg.Sev. 5.55 6.71 6.79 5.83 7.13 5.08
Avg.DoGR 62 100 108 68 107 69
systems increasing the risk of exposure.
Disclosed 251 178 162 52 14 74
On the other hand, forewarned users of vulnerable Fixed 223 156 146 80 15 71
2015

systems are forearmed, so they can take additional Avg.Sev. 6.06 6.66 5.75 6.67 6.53 5.12
prevention actions to mitigate risk of exposure. Besides, Avg.DoGR 79 126 101 133 83 58
public awareness of vulnerability usually pressures vendors Disclosed 194 95 107 21 48 0
to find a fix urgently. Fixed 238 156 159 34 48 17
2016

In the paper we investigate gray risk (or post-disclosure Avg.Sev. 5.52 6.79 6.99 7.53 6.78 -
risk) which defines the interval between vulnerability Avg.DoGR 89 131 93 114 138 -
Disclosed 908 406 489 384 164 202
disclosure time and the date when the patch fixing
Fixed 892 406 489 384 164 202
Total

vulnerability becomes available. We assume vulnerability
disclosure time as a date when a vulnerability firstly appears
in CVE with the corresponding CVE-ID assigned to it.
A time of patch issuing should be derived from vendor’s
security bulletins. It is noteworthy that, according to [26]
about 75% of vulnerability descriptions become available in
the NVD database in seven days on average after they are
mentioned in the vendor security bulletins. It means that
NIST implements so called responsible disclosure model by
giving stakeholders a time for the vulnerability to be patched
before publishing the details in NVD [25]. Besides, it was
also reported that different vendors have different median
announcement gaps: Microsoft – 2 days, Oracle and Apple –
5 days, Linux – 10 days, Novell – approximately 12 days.
In our study we calculate days-of-gray-risk (DoGR) for a
particular vulnerability as a period of time between a
vulnerability is initially reported in CVE until it appears in
NVD. In our work we decided not to take into account these
extra few days, mentioned above, during which vulnerability
descriptions propagate from vendor’s security bulletins to the
NVD database. This can result in a slightly pessimistic
estimate of the OS days-of-gray-risk, which, nevertheless,
seems to be more secure than their underestimate.
Avg.Sev. 5.48 7.12 5.94 6.02 5.71 4.82
Avg.DoR 97 124 133 105 107 74
In the table we use the following short pseudonyms for
the operating systems under investigation:
• Ubuntu – Ubuntu Server 12.04;
• Red Hat – Red Hat Enterprise Linux 6;
• Novell – Novell Linux Enterprise Server 11 SP2;
• Windows – Microsoft Windows Server 2012 R2;
• MacOS – Apple MacOS Server 10.8;
• Solaris – Oracle Solaris 11.
Red Hat Enterprise Linux 6 and Oracle Solaris 11 had
been released before the observed period (see Table II).
Other operating systems (Ubuntu Server 12.04, Novell Linux
Enterprise server 11 SP2, Microsoft Windows Server 2012
R2 and Apple Macintosh Server 10.8) were released in the
beginning of 2012. It is worth mentioning that on the date of
the official release some of those operating systems already
had vulnerabilities that earlier had been discovered in
previous OS versions. In particular, Ubuntu Server 12.04
inherited 15 of such vulnerabilities, Red Hat Enterprise
Linux 6 – 46, Novell Linux Enterprise server 11 SP2 – 26
and Oracle Solaris 11 – 13 vulnerabilities.

208
900
RedHat Novell Ubuntu
800
Ubuntu Solaris
700
Windows Ma cOS
600

500 RedHat
Windows
400 Novell

300
Solaris
200
MacOS
100

0
01.01.2012 01.01.2013 01.01.2014 Date 01.01.2015 01.01.2016 31.12.2016

Figure 2. Cumulative number of disclosed vulnerabilities.

Table III presents average vulnerability severity level
(Avg.Sev.) by aggregating CVSS (Common Vulnerability
Scoring System) metrics taken from NVD vulnerability
database as well as reports average days-of-gray-risk value
(Avg.DoGR). During 2012-2016 the largest number of
vulnerabilities (908) was disclosed in Ubuntu, the least
number (164) – in MacOS. The Red Hat, Windows and
Novell operating systems occupy a middle position having
489, 406 and 384 vulnerabilities, respectively. It is worth to
mention that no new vulnerabilities were discovered in
Solaris during 2016 in contrast to all other operating systems.
A cumulative graph of vulnerabilities disclosed during
2012-2016 is depicted in Fig. 2.
The authors of [27] coin a new term ‘forever-day
vulnerability’ defining a publicly disclosed vulnerability that
has not been patched yet and can be hacked any time during
system operation. It is in contrast to ‘zero-day
vulnerabilities’ [23] which are publically undisclosed
120
RedHat
Novell Ubuntu
100
Ubuntu
Solaris
80
Windows
MacOS
60 RedHat
vulnerabilities that some hackers have already discovered
and can exploit.
Using both, the date of vulnerability disclosure and the
date when the OS vendor issues a patch to fix it we can plot
graphs of forever-day vulnerabilities showing how many of
known (already disclosed publicly) but yet unfixed
vulnerabilities existed every day during 2012-2016 in a
particular operating system (see Fig. 3). Any operating
system running with forever-day vulnerabilities is always
vulnerable unless the software vendor issues a patch and a
system administrator installs it.
As the vulnerability disclosure rate significantly higher
than the rate of vulnerability patching, it can happen that an
operating system contains up to several dozens of forever-day
vulnerabilities at a time. Any of these vulnerabilities could be
potentially exploited by hackers to attack the system. Fig. 3
shows that some operating systems have only few days (if
any) of vulnerability free operation per year.
Windows

40
Novell

20

Solaris MacOS
0
01.01.2012 01.01.2013 01.01.2014 Date 01.01.2015 01.01.2016 31.12.2016
Figure 3. Forever-day vulnerabilities.

209
 For instance (see Table IV), during 2012-2016 OS 250
Ubuntu has not had known vulnerability free days at all. Average
Windows and Red Hat have had only 12 and 10 of such days DoGR Ubuntu
respectively. It means that OS users and administrators 225 Windows
RedHat
should understand and accept potential risk of running
Novell
vulnerable system. In addition, Table IV presents a detailed 200 MacOS
statistics of forever-day vulnerabilities for each operating Sun Solaris Solaris
system during 2012-2016. In average, Ubuntu OS had 47 of
such vulnerabilities every day. For OS Windows, Red Hat 175
and Novell this number is close to 30 vulnerabilities. MacOS
and Solaris had the least average number of forever-day
vulnerabilities (12 and 9 respectively). 150

B. Days-of-Gray-Risk for Operating Systems

125
The number of disclosed vulnerabilities is often used as
the major indicator of software insecurity. However, taking
into account how fast software vendors react on 100
vulnerabilities discovered in their products is also important.
Days-of-risk defines a period of time after a vulnerability is
discovered/disclosed and until it is eliminated from a system 75
after patch installation. It is also known as ‘window of-
vulnerability’ or ‘days-of-recess’. In this study we do not
50
take into account possible delays between the times when a
vendor issues the patch and until a user or a system Oracle Solaris
administrator actually installs it. Besides, in many cases it is 25
impossible to identify when exactly a vulnerability was
discovered. Year
0
TABLE IV. FOREVER DAY VULNERABILITIES STATISTICS 1999 2005 2006 2010 2012 2013 2014 2015 2016
Windows

Red Hat

Figure 4. Operating systems average days-of-gray-risk

MacOS
Ubuntu

Solaris
Novell

Forever day
Year

vulnerabilities (dashed lines depict results reported in earlier studies [13-15, 17,22]).

Min 14 2 21 9 0 6 Thus, in our study we focus on investigating days-of-

Max 51 7 52 29 2 33 gray-risk which is a time after a vulnerability is publicly
2012

Average 23 4 35 19 1 14 disclosed through one of vulnerability databases and until a

Std. deviation 8 1 12 7 1 6 vendor issues a patch fixing it.
Vuln. free days 0 0 0 0 102 0
Min 32 1 17 14 0 4
Days-of-gray-risk can be used to compare efforts that
Max 101 32 60 58 41 17 different vendors make to solve security issues and to deliver
security updates fixing vulnerabilities. Fig. 4 shows how the
2013

Average 65 18 36 31 19 10
Std. deviation 18 7 8 12 11 5 average days-of-risk have been changing over the years for
Vuln. free days 0 0 0 0 9 0 different operating systems. It includes information taken from
Min 12 2 10 11 1 1 Table III (for 2012–2016) as well as data reported for earlier
Max 60 41 40 43 26 22 versions of studied OSs in [13, 14, 15, 17, 28] by other
2014

Average 38 16 21 22 13 8 researchers (depicted using dashed lines). For instance,

Std. deviation 8 10 8 6 8 5
Vuln. free days 0 0 0 0 0 0
according to [15] in 1999 Microsoft had an average of 16 days
Min 28 9 17 10 0 2 from vulnerability disclosure to patch. Red Hat spent only 11
Max 82 98 73 57 14 23 days to fix vulnerabilities while Sun proved itself to be very
2015

Average 48 51 37 24 4 12 slow solving security problems in 90 days on average.

Std. deviation 12 26 12 14 5 6 In 2006, as reported in [13, 28], the days-of-gray-risk
Vuln. free days 0 0 0 0 64 0 parameter for Microsoft Windows series of operating
Min 16 0 0 0 0 0 systems (Windows 2000 Professional and Server, Windows
Max 121 97 108 26 48 17
XP, Windows Server 2003) was estimated at 29 in average.
2016

Average 59 50 41 12 18 1
Std. deviation 31 26 40 9 23 4
At the same time, it took Red Hat 107 days to deliver
Vuln. free days 0 12 10 83 216 268 security updates for its Enterprise Linux 2.1, 3.0 and 4.0
Min 12 0 0 0 0 0 while Sun spent 168 days to do the same for any Solaris
Max 121 98 108 58 48 33 version patched in 2006. In addition, it was estimated that
Total

Average 47 29 34 21 12 9 Apple Mac OS X and Novell SUSE Linux Enterprise Server

Std. deviation 23 26 21 12 15 6 and Desktop (versions 8–10) had 46 and 74 days-of-gray-
Vuln. free days 0 12 10 83 391 268 risk respectively.

210
 Figure 4 shows that since 2013 there has been a tendency
towards decreasing the number of days-of-gray-risks. During
last two years average days-of-gray-risk for different
operating systems varies between 75 and 140 days.
Unfortunately, it still means that after vulnerability public
disclosure users of affected operating system are remaining
vulnerable and unprotected against potential hacker attacks
during months and OS vendors know it.
Our work shows that the conclusion by Jeff Jones
expressed in a series of his earlier blog posts [13, 22, 29] that
Windows is the platform exposing users to risks for the
shortest period of time as compared to other OSs is no longer
correct. At the same time, we can see that since Oracle took
ownership of Solaris OS in 2009 it has been reacting on new
vulnerabilities much faster. Only Oracle Solaris
demonstrates the steady drop of days-of-gray-risks over past
years. Note here, that we did not estimate this parameter for
Solaris in 2016 as there were no new vulnerabilities
discovered.
In addition, we have built the probability density
functions (see Fig. 5) defining the relative likelihood for the
C. Vulnerability Severity
Severity is an important characteristic quantifying the
impact of vulnerability on system’s security. NVD has
adopted Common Vulnerability Scoring System (CVSS) to
assign severity scores to software vulnerabilities.
CVSS are calculated based on several metrics that
approximate ease of vulnerability exploitation (possibility of
remote access, access complexity and need for
authentication), vulnerability impact on confidentiality,
integrity and availability and other factors [30]. Scores, as
well as the overall severity rating are ranged from 0 to 10,
with 10 being the most severe. Besides, vulnerability severity
is divided into several qualitative ranges: Low [0.1..3.9],
Medium [4.0..6.9], High [7.0..8.9] and Critical [9.0..10.0].
In addition to Table III which presents the average
vulnerability severity, Table V shows how the number of
vulnerabilities for different severity levels has been changing
over the recent years for different operating systems.
TABLE V. OSS VULNERABILITY SEVERITY STATISTICS
Operating Number of vulnerabilities by severity score

Year
vulnerability to be patched on a particular day after it was System 1 2 3 4 5 6 7 8 9 10
disclosed, that is much more informative than the average Ubuntu 3 4 1 22 12 12 7 1 2
days-of-gray-risk. It allows us to estimate a probability of Windows 2 2 5 1
issuing patch before the specified date or to define Red Hat 3 3 2 8 4 5 4 1 1
2012
Novell 2 3 1 12 2 8 2 1 1
confidence intervals. MacOS 1 1
This shows, for example, that vulnerabilities in the Solaris 2 6 12 13 7 2 4 1
Novell, Ubuntu and Solaris operating systems are usually Total: 10 17 16 56 27 27 19 1 8 5
fixed during the first 30 days after disclosure. The majority Ubuntu 18 14 7 68 26 32 25 1 2 3
of Windows vulnerabilities are patched between 90 and 150 Windows 1 6 6 5 30 9 2
days while Apple usually spends 60..90 or 120..150 days to Red Hat 10 3 1 20 10 18 7 2
2013

issue security updates for MacOS. Novell 14 11 6 42 11 19 17 1 3

MacOS 3 9 3 18 7 16 3 1
0.35 Solaris 3 3 18 1 2 3 1
p(t) Total: 48 40 18 172 61 92 85 1 13 11
Ubuntu Ubuntu 4 12 5 63 37 19 36 2 10
Windows Windows 3 3 1 9 6 6 16 1 17 2
0.3
Red Hat 1 2 1 12 15 4 17 2 18
RedHat
2014

Novell 2 9 3 53 14 12 15 2 19
Novel MacOS 1 1 3 2 21 3 1 8
0.25 MacOS Solaris 1 4 1 13 6 6 6
Solaris Total: 11 31 12 153 80 68 93 1 24 57
Ubuntu 5 9 13 51 43 38 60 2 13 17
0.2 Windows 3 25 2 11 9 12 66 48 2
Red Hat 3 9 13 36 23 25 42 2 9
2015

Novell 7 1 9 5 3 10 5 12
MacOS 2 4 7 1
0.15
Solaris 5 4 6 20 14 9 16
Total: 18 54 35 127 94 91 201 2 68 41
Ubuntu 5 16 2 60 37 32 25 10 7
0.1 Windows 3 7 2 14 5 7 23 29 5
Red Hat 2 1 23 12 21 8 1 32 7
2016

Novell 4 1 1 7 5 3
0.05 MacOS 3 11 8 1 6 17 2
Solaris
Total: 8 28 5 112 63 62 69 1 93 24
t, days Ubuntu 35 55 28 264 155 133 153 3 28 39
0
Windows 9 35 6 40 28 30 137 1 108 12
30 60 90 120 150 180 210 240 270 300 330 360 >360
Sum-total

Red Hat 17 19 18 99 64 73 78 1 37 37
Figure 5. Probability density functions of OS days-of-gray-risk. Novell 18 30 11 120 33 43 51 0 14 38
MacOS 5 14 4 33 17 42 19 0 19 11
Solaris 11 17 19 64 28 19 29 1 0 1
TOTAL: 95 170 86 620 325 340 467 6 206 138

211
35% In our research we have checked a widespread hypothesis
Percentage of
that software vendors make more efforts on fixing the most
vulnerabilities
critical vulnerabilities firstly. However, a diagram on Fig. 7
30%
Ubuntu
shows that days-of-risk metric does not actually depend on
vulnerability severity. To some extent it seems to be true for
Windows the Red Hat operating system, however, developers of all the
25%
RedHat rest operating systems spend considerably more time in
Novell average on fixing critical vulnerabilities as compared to the
20% least severe ones.
MacOS
Solaris D. The Most Common OSs Vulnerabilities
15%
NVD classifies all vulnerabilities using the CWE
scheme. The Common Weakness Enumeration (CWE) is a
10% formal list of software weakness types proposed by MITRE
Corporation (https://cwe.mitre.org/).
The top ten vulnerability types discovered in different
5% operating systems is presented in Table VI. About 70 percent
of vulnerabilities in Oracle Solaris are marked as
Vulnerability severity
0%
‘Uncategorized’. Thus we took them out of consideration.
1 2 3 4 5 6 7 8 9 10 The most common OS vulnerabilities by the CWE types
(sorted by prevalence) are:
Figure 6. OS vulnerabilities distribution by CVSS severity level. CWE-119 – Improper restriction of operations within the
bounds of a memory buffer using lacks of certain
Vulnerabilities in Oracle Solaris are the least critical. programming languages (often C and C++) that do not
Their average severity is 4.82 (see Table III). The most control bounds for the memory buffer that is being
severe vulnerabilities have been discovered in OS Microsoft addressed. Vulnerabilities of the CWE-119 type usually
Windows (average severity is 7.12) and Novel (the average cause arbitrary code execution, altering the intended control
severity is 6.02). flow, reading protected information or system crash;
Fig. 6 shows the percentage of vulnerabilities for CWE-264 – Weaknesses and mistakes in permissions,
different severity levels. A number of critical vulnerabilities privileges, and access controls;
[9.0..10.0] disclosed in the Microsoft Windows OS is equal CWE-20 – Improper input validation which may result in
to almost 30% of total. At the same time, the quantity of such altered control flow, arbitrary code execution or illegal
vulnerabilities for other operating systems is less than 19%. access to and control of resources;
The fewest percentage of critical vulnerabilities (less than CWE-200 – Information intentional or unintentional
1%) was observed in Solaris. exposure to an actor that is not explicitly authorized to have
access to that information;
300
CWE-399 – Improper management of system resources,
Average
DoGR Ubuntu Windows e.g. memory allocation or reallocation;
CWE-189 – Numeric errors related to improper
250 RedHat Novell
calculation or conversion of numbers;
MacOS Solaris CWE-362 – Concurrent code execution using shared
resource with improper synchronization also knows as Race
200
Condition;
CWE-310 – Cryptographic issues including missing
encryption of sensitive data or key management errors;
150
CWE-94 – Improper control of code generation also
known as Code Injection which often happens when
100
software allows a user's input to contain code syntax.
CWE-59 – Improper link resolution before file access
that allows an attacker to traverse the file system to
50
unintended locations and read/overwrite the contents of
unexpected files.
Figure 8 depicts the percentage of OS vulnerabilities
0 distributed by different CWE types and shows how it had
1 2 3 4 5 6 7 8 9 10 been changing during 2012-2016. The first three types of
Vulnerability severity vulnerabilities (CWE119, 264 and 20) have been dominating
Figure 7. Average days-of-gray-risk depending on vulnerability severity. over these years. In 2016 their contribution to the total
number of discovered vulnerabilities has exceeded 70%.

212
 TABLE VI. THE MOST COMMON OS VULNERABILITY TYPES It is worth noting that numeric errors caused by incorrect
Vulnerability Percentage of vulnerabilities by CWE types calculation or conversion of numbers (CWE-189) affect not
Type Ubuntu Windows Red Hat Novell MacOS Solaris Total* only system security but also system dependability and
CWE-119 22.62 12.17 15.58 15.64 23.90 6.99 17.98 trustworthiness. This type of errors still remains quite
CWE-264 11.95 30.17 7.67 13.97 18.87 0.54 16.53 typical, sometimes causing costly (or, even embarrassing)
CWE-20 11.14 14.60 6.77 12.29 19.50 4.30 12.86 system failures. In particular, the authors of [31] found out
CWE-200 7.54 12.65 6.09 8.66 12.58 1.61 9.51 that more than 700 research papers reporting on various
CWE-399 7.42 3.16 3.39 7.82 2.52 4.30 4.86 strands of the genomic research over the 10-year period are
CWE-189 7.54 1.70 5.19 6.98 2.52 5.91 4.79
CWE-310 1.97 1.22 1.13 3.35 5.66 - 2.67
riddled with errors due to an erroneous conversation of some
CWE-362 2.90 1.46 2.03 5.31 0.00 - 2.34 gene symbols (e.g. MARCH1 or 2310009E13) to date and
CWE-94 0.46 5.84 0.45 0.00 0.00 - 1.35 numbers in Excel spreadsheets.
CWE-59 0.70 0.24 1.13 0.28 0.00 1.08 0.47
Others 8.12 11.44 9.93 2.79 6.92 3.76 7.84
E. Common and Group Vulnerabilities
Uncategorized 17.63 5.35 40.63 22.91 7.55 71.51 18.81 The most dangerous vulnerabilities are those discovered
*Taking out of consideration Solaris vulnerabilities in more than one operating system. A reason why the same
vulnerability is discovered in several OSs is explained by
CWE-119 weaknesses (e.g. CVE-2016-7277, CVE-2016- using common vulnerable components (system libraries,
4658 or CVE-2016-4598) often allow remote attackers to third party software components, OS kernels, etc.).
execute arbitrary code, read protected data or cause a denial More often group vulnerabilities are discovered in
of service via a crafted document viewed by victim on the different releases of the same OS or in a family of related
infected web-page (e.g. .jpeg image or .xml file) or operating systems, e.g. BSD Unix (OpenBSD, FreeBSD,
downloaded from the Internet and opened on his/her NetBSD) or Linux (Red Hat, CentOS, Novell, Ubuntu), etc.
computer (e.g. .doc/.pdf documents or media files). However, sometimes hackers and security analysts
CWE-264 weaknesses usually mean implementation discover vulnerabilities that are common for even different
mistakes in permissions, privileges, and access control OS families. For example, the CVE-2008-4609 vulnerability
mechanisms. As a result, vulnerable software cannot caused denial-of-service attack for a variety of operating
properly identify session reuse (CVE-2016-3840), bypasses systems and their versions, including Linux, BSD Unix,
check for the access, read or write permissions (CVE-2016- Microsoft Windows, Cisco IOS and possibly many others
2416 or CVE-2016-6536) or relies on client-side [32, 33]. The vulnerability manipulated the state of
authorization that can be easily bypassed via certain changes Transmission Control Protocol (TCP) connections exploiting
in local files (CVE-2015-5989). an algorithmic error in protocol implementation in various
Improper input validation (CWE-20) is a parent of other operating systems. A remote attacker was able to cause
widespread vulnerability types including command injection connection queue exhaustion by flags manipulation in the
(CWE-77), cross-site scripting (CWE-79) and SQL injection TCP header of crafted network packets sent to a victim-
(CWE-89). computer.
Figure 9 shows common vulnerabilities distributed
100% 21 64 6 7 4 1 3 between the Ubuntu, Novell and Red Hat operating systems
14 11 6
6 24 4 9 CWE-59 during 2012-2016. Forty seven of them were disclosed in all
9 8
90% 5 19 20 36 three operating systems. Besides, there were 3 groups of
45 25 CWE-94 vulnerabilities shared between the following pairs: Ubuntu
80% 20 24 57
and Novell – 241, Red Hat and Ubuntu – 65, and Novell and
38 46 CWE-310
45 Red Hat – 36.
70%
17 CWE-362
The numbers in brackets correspond to those
59 23 38
60% 73 vulnerabilities observed in Linux kernels (the NVD database
CWE-189 distinguishes between vulnerabilities observed in Linux-
13 based operating systems and Linux-kernels). Thus, Fig. 9
50% 74 67
62 convincingly demonstrates that the largest number of
CWE-399
40% 17 95 common and group vulnerabilities shared between the
CWE-200 Ubuntu, Novell and Red Hat operating systems are those
30% 84 discovered in the Linux kernels (versions 3.2.x, 3.0.x and
81
20 CWE-20 2.6.32) used by them. Red Hat and MaɫOS only share the
20% 106 CVE-2013-1824 vulnerability in the PHP SOAP parser
132 CWE-264 which allows a remote attacker to gain unauthorised access
10% 101 to arbitrary files of operating systems.
21 61
CWE-119 The number of vulnerabilities shared by two or more
0% operating systems can be used as a measure of diversity
2012 2013 2014 2015 2016 between them [10]. Software diversity [34, 35, 36] has been
Figure 8. Number of vulnerabilities of different CWE types used as a major fault and intrusion-tolerance mechanism to
distributed by years. design safety-critical computer systems.

213

Ubuntu Novell
241 (216)
544 50
57 (37)
65 (37) 36 (14)
331
Red Hat
Figure 9. Number of individual, group and common vulnerabilities shared
by Linux family of operating systems (Ubuntu, Novell and Red Hat).
Thus, vulnerability databases (the NVD database in
particular) can help in determining the most diverse software
products.
Our analysis shows that the results reported in [10]
should be further verified as the authors may not have
considered common and group vulnerabilities observed in
Linux kernels.
V. CONCLUSION AND LESSONS LEARNT
The paper presents a retrospective vulnerability analysis
of popular enterprise operating systems: Ubuntu Server
12.04, Red Hat Enterprise Linux 6, Novell Linux Enterprise
Server 11 SP2, Microsoft Windows Server 2012 R2, Apple
MacOS Server 10.8 and Oracle Sun Solaris 11.
The statistics discussed in the paper have been collected
by querying a specially developed MySQL database which
aggregates the XML data feeds provided by the CVE
(www.cve.mitre.org) and NVD (web.nvd.nist.gov)
vulnerability repositories.
A significant growth of the total number of
vulnerabilities discovered in modern operating systems as
well as the general tendency toward increasing their severity
demonstrate the serious security challenges and risks that OS
developers and users face. It is very important to understand
that the crucial parameters affecting system security are not
only the total number of vulnerabilities disclosed in a
particular software product and their severity but also, so
called, days-of-risk which show how fast software vendors
issue patches fixing disclosed vulnerabilities. Our study
shows that the average days-of-risk for the investigated
operating systems varies from 83 days for Oracle Solaris up
to ɞɨ 135 days for Red Hat.
It is worrying that as the paper shows, the rate with
which software developers issue security updates in general
does not depend on vulnerability severity. Average days-of-
gray-risk for the most critical vulnerabilities remains even
higher than the one calculated for vulnerability of the lowest
severity (117 vs 100 days). This uncovers worrying
shortcomings in the policies for developing security updates
adopted by OS vendors, as well as, in the maintenance
management processes they run.
214
The number of OS vulnerabilities that remain unpatched
during a considerable period of time is growing. The increase
of days-of-gray-risk and the raise of a number of forever-day
vulnerabilities threaten security and dependability of
computer systems.
Users and system administrators should be aware of the
fact that the operating systems they use typically have up to
several dozens of known but yet unfixed vulnerabilities (our
study reports 25 forever-day vulnerabilities in average for the
investigated operating systems for every day during
2012-2016).
It is worth noting that since the beginning of 2016 the
NVD database has reported 31 vulnerabilities (not only in
the studied OSs) as those that become possible because of an
incorrect fix of some previous vulnerabilities. For instance,
the CVE-2016-2550 vulnerability in Linux kernel before 4.5,
causing denial of service, was introduced as a result of the
incorrect fix for the CVE-2013-4312 vulnerability. Besides,
196 vulnerabilities have been reported as the ones that
existed because of incomplete fixes (e.g. CVE-2016-10088).
This means that waiting for and fully relaying on security
updates provided by software vendors is not a panacea. The
implementation of the defence-in-depth principle by applying
the layered security mechanisms to cope with the forever-day
vulnerabilities is of a great importance for modern computer
systems.
One specific aspect that the paper studies is the
vulnerabilities that were discovered in more than one
operating systems. Such vulnerabilities that are common for
different operating systems and even different OS families
can lead to large-scale hacker attacks and virus epidemics.
They also seriously complicate the development of intrusion-
tolerant computer systems based on OS diversity.
The results presented in the paper show that numerous
vulnerabilities in operating systems cause significant security
threats. This calls for implementing complex and in-depth
defense mechanisms that incorporate antivirus software,
firewalls, security scanners, intrusion detection systems and
other solutions in a way that makes them aware of the recent
and current OS vulnerabilities.
Hopefully, this work will make OS vendors to pay much
more attention to improving security of their products, which
might require significant changes in their software
development and maintenance processes. Our experimental
work supports our claim that decreasing days-of-risk and
reducing a number of forever-day vulnerabilities is one of
the main challenges in building secure operating systems.
We believe that the methodology presented in this paper
can be used to create a public service reporting forever-day
vulnerabilities (and other crucial vulnerability statistics)
observed daily in each OS and other products to help system
administrators in understanding security risks and in taking
appropriate protective actions.
ACKNOWLEDGMENTS
Alexander Romanovsky is partially supported by the
EPSRC/UK STRATA platform grant. Anatoliy Gorbenko
and Olga Tarasyuk are partially supported by the TEMPUS
SEREIN grant.
 REFERENCES
[1] B. Clark, “Hackers take hospital offline, demand $3.6m ransom,”
[Online]. Available: http://thenextweb.com/insider/2016/02/15/
hackers-take-hospital-offline-demand-3-6m-ransom/.
[2] C. Williams, “Passengers ride free on SF Muni subway after
ransomware infects network, demands $73k,” [Online]. Available:
http://www.theregister.co.uk/2016/11/27/san_francisco_muni_ranso
mware/.
[3] MITRE Corporation, “Common Vulnerabilities and Exposures.
Terminology,” [Online]. Available: https://cve.mitre.org/
about/terminology.html.
[4] Microsoft Inc., “Description of Software Update Services and
Windows Server Update Services changes in content for 2016,”
2016. [Online]. Available: https://support.microsoft.com/en-
us/help/3215781/description-of-software-update-services-and-
windows-server-update-services-changes-in-content-for-2016.
[5] National Vulnerability Database, “Vulnerability Summary for CVE-
2016-7256,” [Online]. Available: https://web.nvd.nist.gov/view/
vuln/detail?vulnId=CVE-2016-7256.
[6] MITRE Inc., “CVE-2016-7256,” [Online]. Available:
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-
7256.
[7] Microsoft Inc., “Microsoft Security Bulletin MS16-132 - Critical,”
[Online]. Available: https://technet.microsoft.com/en-us/library/
security/ms16-132.aspx.
[8] National Vulnerability Database, “Vulnerability Summary for CVE-
2014-0160,” [Online]. Available: https://web.nvd.nist.gov/view/
vuln/detail?vulnId=CVE-2014-0160.
[9] Z. Schreuders, T. McGill and C. Payne, “The state of the art of
application restrictions and sandboxes: A survey of application-
oriented access controls and their shortfalls,” Computers and
Security, vol. 32, pp. 219-241, 2013.
[10] M. Garcia, A. Bessani, I. Gashi, N. Neves and R. Obelheiro, “OS
Diversity for Intrusion Tolerance: Myth or Reality?,” in IEEE/IFIP
41st Int. Conf. on Dependable Systems & Networks (DSN’2011),
2011.
[11] S. Frei, M. May, U. Fiedler and B. Plattner, “Large-scale
vulnerability analysis,” in SIGCOMM Workshop on Large-Scale
Attack Defense, 2006.
[12] M. Shahzad, M. Zubair Shafiq and A. Liu, “A large scale
exploratory analysis of software vulnerability life cycles,” in 34th
Int. Conf. on Software Engineering (ICSE '12), 2012.
[13] J. Jones, “Days-of-risk in 2006: Linux, Mac OS X, Solaris and
Windows,” 2006. [Online]. Available:
http://www.csoonline.com/article/2136935/data-protection/days-of-
risk-in-2006---linux--mac-os-x--solaris-and-windows.html.
[14] A. Gorbenko, O. Tarasyuk, V. Kharchenko and A. Romanovsky,
“Using Diversity in Cloud-Based Deployment Environment to
Avoid Intrusions,” Software Engineering for Resilient Systems, no.
LNCS 6968, p. 145–155, 2011.
[15] J. Reavis, “Linux vs. Microsoft: Who Solves Security Problems
Faster?,” 2000. [Online]. Available: http://www.reavis.org/
research/solve.shtml.
[16] P. Edmonds, “When It Comes to Protection from Vulnerabilities,
Process Trumps “Many Eyes”,” 2007. [Online]. Available:
https://technet.microsoft.com/en-us/library/cc512608.aspx.
[17] A. Patrizio, “Report Says Windows Gets The Fastest Repairs,”
2007. [Online]. Available: http://www.internetnews.com/security/
article.php/3667201.
[18] OSVDB, “Rebuttal: Dark Reading’s “9” Sources for Tracking New
Vulnerabilities,” 2016. [Online]. Available:
https://blog.osvdb.org/2016/10/26/rebuttal-dark-readings-9-sources-
for-tracking-new-vulnerabilities/.
215
[19] M. Cheung, “Market Share Analysis: Server Operating Systems,
Worldwide, 2015: Gartner report.,” 2016. [Online]. Available:
https://www.gartner.com/doc/3326217/market-share-analysis-
server-operating.
[20] P. Tsai, “Server Virtualization and OS Trends,” 2016. [Online].
Available:
https://community.spiceworks.com/networking/articles/2462-server-
virtualization-and-os-trends.
[21] W3Techs, “Usage of operating systems for websites,” 2017.
[Online]. Available:
https://w3techs.com/technologies/report/operating_system.
[22] J. Jones, “Basic Guide to Days of Risk,” 2007. [Online]. Available:
http://www.csoonline.com/article/2136934/data-protection/basic-
guide-to-days-of-risk.html.
[23] L. Bilge and T. Dumitras, “Before we knew it: An empirical study
of zero-day attacks in the real world,” in ACM Conference on
Computer and Communications Security, Raleigh, NC, 2012.
[24] B. Barth, “Lag between a bug's first disclosure and its inclusion in
national database can put companies at risk”,” 2017. [Online].
Available: https://www.scmagazine.com/lag-between-a-bugs-first-
disclosure-and-its-inclusion-in-national-database-can-put-
companies-at-risk/article/667398/.
[25] A. Hahn and M. Govindarasu, “Cyber vulnerability disclosure
policies for the smart grid,” in IEEE Power and Energy Society
General Meeting, San Diego, USA, 2012.
[26] B. Ladd, “The Race Between Security Professionals and
Adversaries,” 2017. [Online]. Available:
https://www.recordedfuture.com/ vulnerability-disclosure-delay/.
[27] D. Goodin, “Rise of “forever day” bugs in industrial systems
threatens critical infrastructure,” 2012. [Online]. Available:
http://arstechnica.com/business/2012/04/rise-of-ics-forever-day-
vulnerabiliities-threaten-critical-infrastructure/.
[28] M. Oiaga, “Recount: Windows Still Safest, Tops Mac OS X, Linux
and Sun Solaris. But are statistics a true measure of security?,”
2007. [Online]. Available: http://news.softpedia.com/news/Recount-
Windows-Still-Safest-Tops-Mac-OS-X-Linux-and-Sun-Solaris-
57433.shtml.
[29] J. Jones, “2006 Client OS Days of Risk,” 2007. [Online]. Available:
https://blogs.microsoft.com/microsoftsecure/2007/06/18/2006-
client-os-days-of-risk/.
[30] Forum of Incident Response and Security Teams, “Common
Vulnerability Scoring System, V3 Development Update,” 2015.
[Online]. Available: https://www.first.org/cvss.
[31] M. Ziemann, Y. Eren and A. El-Osta, “Gene name errors are
widespread in the scientific literature,” Genome Biology, vol. 17,
no. 177, 2016.
[32] National Vulnerability Database, “Vulnerability Summary for CVE-
2008-4609,” 2008. [Online]. Available: https://web.nvd.nist.gov/
view/vuln/detail?vulnId=CVE-2008-4609.
[33] Cisco Systems, “TCP State Manipulation Denial of Service
Vulnerabilities in Multiple Cisco Products,” 2009. [Online].
Available: https://tools.cisco.com/security/center/content/
CiscoSecurityAdvisory/cisco-sa-20090908-tcp24.
[34] B. Randell, “System Structure for Software Fault Tolerance,” IEEE
Transactions on Software Engineering, vol. 1, no. 2, pp. 221-232 ,
1975.
[35] A. Avizienis, “The N-Version Approach to Fault-Tolerant
Software,” IEEE Transactions on Software Engineering , vol. 11,
no. 12, pp. 1491-1501, 1985.
[36] B. Littlewood and L. Strigini, “Redundancy and diversity in
security,” in 9th European Symposium on Research Computer
Security ( ESORICS'2004), LNCS 3193, 2004.