Security and Risk Management Strategies

In-Depth Research Field Research Results - Participants

Speak

Field Research: Security Metrics

Version: 1.0, Dec 03, 2009

AUTHOR(S):
Phil Schacter
(pschacter@burtongroup.com)

Additional Input:
Eric Maiwald, Ramon Krikken

TECHNOLOGY THREAD:
Security Concepts, Techniques, and Approaches

129814
Table Of Contents
Business Context.............................................................................................................................................................3
The Participants Speak....................................................................................................................................................4
What We Discovered...................................................................................................................................................... 5
Security Metrics.......................................................................................................................................................... 5
Maturity of Metrics Programs.................................................................................................................................5
Qualitative Measures...............................................................................................................................................5
Technical Metrics....................................................................................................................................................6
Reporting to Management.......................................................................................................................................6
Tools and Automation of Metrics........................................................................................................................... 6
How Metrics Are Used........................................................................................................................................... 8
Related Research and Recommended Reading...........................................................................................................8
Author Bio ......................................................................................................................................................................9

2
BURTON GROUP 7090 Union Park Center Suite 200 Midvale · Utah 84047 · P 801.566.2880 · F 801.566.3611 · www.burtongroup.com
Business Context
Effective governance of system and information security policies, programs, and support organizations is critical
to management's responsibility to maintain and protect the business within the scope of applicable laws and
practices. An information system security program enables an organization to accept or manage risk related to
systems technology, communication technology, people, and processes required to electronically conduct
business. The program also helps the organization ensure system integrity and availability, as well as appropriate
system security accountability. An effective information security program provides significant benefits for the
business and can create a competitive advantage. By enforcing privacy and confidentiality and avoiding a
damaging security incident (e.g., having employee or customer information inappropriately shared, lost, or
exposed), the business sustains its market value, its reputation, and its commitment (contractual or implied) to
shareholders, customers, partners, and suppliers.

3
BURTON GROUP 7090 Union Park Center Suite 200 Midvale · Utah 84047 · P 801.566.2880 · F 801.566.3611 · www.burtongroup.com
The Participants Speak
IT security organizations take different approaches to measuring the effectiveness and maturity of their security
programs. The statements in this section represent just a few of the diverse opinions gathered by Burton Group
analysts during a field research project conducted from February through April 2009.
One CISO's comments on the perceived futility of efforts to measure the success of security programs:
• “No one has ever been able to measure security—executives only care about the incidents that impact the
business.”
• “Quantitative measurements of security can't be done.”
• “Success is really binary—it was prevented or not. If there's no impact, then no one cares that it was
prevented.”
• “If the business continues to run, then that's success.”
Some interview comments on how metrics are used:
• Metrics are used to raise awareness of issues and to bring projects along.
• Operational security metrics are reviewed during weekly Operations meeting.
• Security is part of the report card for each business unit. Contract risk and profitability are also part of the
report card.
• Security metrics are compared to peers in the industry, as well as to other business units.
• One CISO reports to executives on the status of how well policies are implemented in the business lines, and
how well security controls the policies.
On the use of frameworks, automation, and dashboards:
• One CISO uses a spreadsheet to report operational metrics.
• Another CISO plans to implement a risk warehouse and dashboard based on internal data and processes.
• Other organizations are using security scorecards.

4
BURTON GROUP 7090 Union Park Center Suite 200 Midvale · Utah 84047 · P 801.566.2880 · F 801.566.3611 · www.burtongroup.com
What We Discovered
This document covers a subset of the findings of Burton Group's field research on security governance and
security programs that was conducted over the period February through April 2009. The specific focus of this
document is to present information on how the organizations conduct security metrics programs that seek to
measure the effectiveness of security and risk programs.

Security Metrics
The maturity of security metrics programs varies considerably, and this observation was validated in
conversations with CISOs and IT security managers during this field research. A range of technical and
management metrics are tracked for internal review, and to report to executive level security governance councils.
An increasing number of organizations are building more formal metrics programs in order to better understand
how well their security programs are performing and whether security is improving.

Maturity of Metrics Programs

IT security groups have struggled for many years to identify meaningful metrics to demonstrate that progress is
being made toward a more secure organization. Even organizations with relatively mature metrics programs are
looking to improve their process and level of automation.
Over the past three years, more organizations are seeing the value of formal security metrics programs. One
organization uses a maturity model to assess the level of security in each business unit. Another organization has a
formal process where a metrics review board evaluates new proposed metrics before they are published and relied
on. In yet another organization, the metrics program was only recently formalized in 2008.
But other organizations acknowledge that their security metrics programs are weak and need to improve. One
interviewee acknowledged that the security and risk group did not know how to measure itself. Another CISO
indicated that there were no defined metrics that his team was measured by. But another interviewee was clearly
trying to identify compelling metrics to measure the security program against; unfortunately, so far he hasn't been
successful.
Some organizations conduct periodic formal and informal benchmarking with peers in the industry to compare
security programs and operational security metrics. One CISO establishes goals, based on a survey conducted
every three years by an industry group, for how the organization should match up against peers in the industry.
Another CISO prepares a similar report based on an industry security benchmark, which is reported up to the
CIO, senior business executives, CEO, and the board.

Qualitative Measures
For many organizations, the effectiveness of the security program and IT security group is based on subjective
factors, not quantitative metrics. One interviewee contrasted the natures of quantitative and qualitative analysis by
saying quantitative models are crude due to the difficulty in getting accurate likelihood of event occurring data,
while qualitative maps to industry best practices. One CISO commented that he was measured based on how
much gets done and “how others feel about it.” Another CISO commented that he was measured on intangibles,
with less fault being found on specifics.
While it's common for organizations to measure the success of specific security projects, there's no clear metric
that can assess the overall effectiveness of the security program. In the words of one interviewee, “if the business
continues to run, then that's success.” For another CISO, the lack of recent security incidents is viewed as a sign
that security is doing its job well. In another organization, the interviewee stated that the ultimate measurement is
successfully keeping “bad people” out. At a high level, the goals of one organization's metrics program are to
minimize audit findings of material weaknesses and prevent disclosure of information prohibited by regulation.

5
BURTON GROUP 7090 Union Park Center Suite 200 Midvale · Utah 84047 · P 801.566.2880 · F 801.566.3611 · www.burtongroup.com
Technical Metrics
Many security organizations track metrics that reflect numbers reported by various security tools, that result from
operating security mechanisms, or that reflect the progress of the security program in a number of areas, such as:
• Number of patched systems
• How quickly patches are deployed
• Status of mitigating audit findings
• Results of vulnerability scans
• Number of vulnerabilities per line of code
• Number of laptops with encrypted hard drives
• Number of viruses detected/blocked (by region)
• Volume of spam e-mail blocked
• Percentage of systems running latest antivirus signatures
• Closure rate on security-related trouble tickets
• Number of lost laptops
• Number of e-mail policy violations (detected by data loss prevention [DLP] product)
• Number of incidents where data is disclosed
• Rate of resolution of compliance exceptions
• Number of user accounts provisioned and time before available for use
Generally, this kind of metric is used internally by the IT security group to assess whether security programs are
effective and making progress, but it is not rolled up and reported to senior management.

Reporting to Management
Finding metrics that are meaningful to senior executives is a challenge for many CISOs. One CISO commented
that he doesn't want to bother people with numbers because it is not effective in reporting to management.
Another CISO said, “No one has ever been able to measure security—executives only care about the incidents
that impact business.” At one organization, the CISO does not report metrics upward, while in another case the
interviewee commented that management doesn't care about operational metrics as long as there are no security
incidents. Similarly, another interviewee stated “The CEO isn't interested in security incident metrics.” Another
CISO explained that management did not want to see details of tool metrics.
Some of the measures that IT security groups report to senior management include:
• Number of security assessments performed
• Exceptions to compliance policies
• Application risk scores
• Internal and external audit findings
• Security awareness attestations, surveys, and test results
• Incidents and related costs
• Satisfaction surveys
• Service levels in responding to requests (e.g., add user, reset password, and modify firewall rule)
• Compliance with internal and external service level agreements (SLAs)
• Security group budget and accuracy of project cost estimates
• Status of active security projects
• Payment Card Industry Data Security Standard (PCI DSS) compliance status and dates met

Tools and Automation of Metrics

6
BURTON GROUP 7090 Union Park Center Suite 200 Midvale · Utah 84047 · P 801.566.2880 · F 801.566.3611 · www.burtongroup.com
 As security metrics programs mature, data is gathered from operational sources and represented in spreadsheets,
dashboards, and custom tools that build on commercial frameworks (e.g., Archer Technologies' SmartSuite). One
organization found that efforts to leverage existing operational data often revealed process breakdowns that
needed to be fixed before proceeding. One CISO cautioned “Only gather the data you plan to use.”
Existing trouble ticket systems are one source for useful metrics on the security team's handling of service
requests and incidents. Also, security tools offer reporting options that can provide information on activity that
can manually be compiled into a spreadsheet, report, or dashboard. One organization tracks 70 to 80 operational
metrics in a dashboard view of a spreadsheet, where each number is assigned to a green, yellow, or red status.
Another organization maintains the current status of its metrics online for ad hoc access by authorized internal
stakeholders (see Figure 1).

Figure 1: Example of Operational Security and Compliance Dashboard

7
BURTON GROUP 7090 Union Park Center Suite 200 Midvale · Utah 84047 · P 801.566.2880 · F 801.566.3611 · www.burtongroup.com
How Metrics Are Used
Security organizations use metrics to manage internal projects, track trends and progress toward personal and
team goals, and report status to management and other parts of the organization. In some cases metrics are used as
one component of performance reviews, and when determining individual and team bonus or merit-based
compensation.
Different metrics are periodically reviewed by different groups within the organization. For example, operational
security metrics are reviewed in a weekly operations meeting at one of the interviewed organizations. CISOs also
report security and compliance metrics and project status to CIOs or risk governance councils on a monthly or
quarterly basis, depending on the organization and its governance model. In some cases, these metrics are rolled
up by the CIO and presented to senior business management, the CEO, and the board. One CISO commented that
there is a performance objective to improve the percentage compliance of IT systems with required controls. In
another organization, each business unit is issued a report card that includes grades for contract risk, profitability,
and security.
Security metrics are also used by some organizations to assess individual and team performance. Security team
objectives and bonuses may be linked to improvement in specific security metrics or to the security dashboard. In
one organization, the CIO reports progress on IT objectives to the board, which gives both objectives and metrics
high visibility. In another example, the security group is measured based on level of service to requests, such as
provisioning of new users and accounts.

Related Research and Recommended Reading

For additional information on IT and security metrics topics, refer to the following Burton Group documents:
• “Security Key Performance Indicators” (overview)
• “Security Metrics: Horses for Courses” (overview)
• “Introduction to Key Risk Indicators” (Methodologies and Best Practices document)
• “Thinking Strategically About Security Metrics” (overview)
• “Using Metrics Effectively: Proving and Improving the Business Value of IT” (Perspective document)
• “You Manage What You Measure” (Perspective document)

8
BURTON GROUP 7090 Union Park Center Suite 200 Midvale · Utah 84047 · P 801.566.2880 · F 801.566.3611 · www.burtongroup.com
Author Bio
Phil Schacter
Vice President and Service Director
Emphasis: web security, enterprise security, network security, identity management, provisioning
Background: Phil Schacter is a Vice President and service director for Burton Group Security and Risk
Management strategies. He covers enterprise security, security governance, network security, and security reference
architectures. Prior to joining Burton Group, Phil has worked in the network technology industry with experience in
mainframe network applications, network-delivered services, distributed messaging systems, security policy and
architecture, security consulting, and identity management systems. With 35 years of industry experience, Phil has
designed and developed network applications, managed network services, and messaging product lines. He has
worked on standards groups and authored reports on IT technologies and architecture, security, and identity
management topics.

Copyright 2009 Burton Group. ISSN 1048-4620. All rights reserved. All product, technology and service names are trademarks or service marks
of their respected owners. See Terms of Use and publishing information at http://www.burtongroup.com/AboutUs/TermsOfUse.aspx

9
BURTON GROUP 7090 Union Park Center Suite 200 Midvale · Utah 84047 · P 801.566.2880 · F 801.566.3611 · www.burtongroup.com