Documente Academic
Documente Profesional
Documente Cultură
Executive summary
Volumes of data are expanding rapidly, and
effectively harnessing the data volumes generated by organisations today brings both
significant gains as well as challenges. In particular, big data security sets bring specialised
challenges owing to the nature of the information that is produced, which must be stored in
a sequential, time-stamped manner and which
must be stored in its raw, unchanged format
that proves its integrity; in other words, it has
not been tampered with.
This requires the use of specialist tools. Traditional database and warehousing technologies
do a good job of handling operational data,
but were not designed with the specificities of
security data in mind. They are also based on
an architecture that is difficult to scale effectively to meet the challenges of huge data sets,
such as the huge volumes and constant flow
of security event information from systems
throughout the organisation.
Even where the back end technology has been
built with the needs of big data security event
information in mind, effective analysis of that
information requires a security intelligence
platform be integrated at the front end so that
information flows freely in an uninterrupted
manner between all parts of the system. Many
vendors offer partial solutionseither the
back end technology or the front end security
intelligence platform. This creates multiple integration and management challenges that are
not only a challenge in the upfront implementation, but also in the ongoing management
and extensibility of the system as new data
sources are added. Few organisations have
the resources or budget available to effectively
overcome the challenges of integrating disparate technologies. Far better is to look for
a system that was built from the ground up as
one integrated system, designed specifically
for the intricacies involved with security data.
Fast facts
Enabling
experimentation to discover
needs, expose variability and improve performanceuse data to analyse variability in performance and understand the root
causes.
Replacing/supporting
Gartner Group
As harnessing big data has become a strategic priority for many organisations, many have focused primarily on operational data in an effort
to improve business performance and efficiency. However, the value
of big data extends way beyond this and has enormous application for
improving security within organisations. Whilst the use of big data for
security currently lags behind its use for operational purposes, the
Gartner Group estimates that 40% of enterprisesled by the banking, insurance, pharmaceutical and defence industrieswill actively
analyse patterns using data sets of at least 10 terabytes in order to flag
potentially dangerous activity by 20165.
The ability to harness big data sets from throughout the organisation
provides valuable insight into the security intelligence buried within very
large volumes of complex data from disparate data sources that will
boost the organisations ability to understand and manage the risks and
threats that it faces (See Figure 1). By analysing and correlating this
information to provide greater context, organisations will be afforded a
much higher level of understanding as to which of their assets are most
vulnerable to attack and where those attacks are most likely to come
from. This will allow them to take more focused and directed action,
prioritising risks against the severity of vulnerabilities faced in order to
take better decisions as to what risks to fix first.
Through advanced analytics and correlation across multiple, voluminous data sets, organisations will also be better able to ward off the
increasingly sophisticated, complex and targeted attacks that are the
reality today. Many of those threats are specifically designed to avoid
defences such as anti malware and intrusion detection systems that
are based on fingerprinting specific patterns associated with known
exploits. Big data analytics will boost an organisations ability to develop
models of what constitutes expected behaviour, which can then be used
as a baseline against which the analysis will be able to discern meaningful patterns and pinpoint behaviour that is abnormal or suspicious
across multiple parts of the network to identify behaviour likely to be
associated with an attack. In this way, organisations will be able to take
a more proactive stance on security; identifying trends, attack profiles
and possible threats, rather than just reacting to each incident as it occurs. They will be in a better position to not only prevent attacks, but
also to gain a better understanding regarding the threats that they face
in order to take more focused and directed action against those threats
and sources of breaches.
One further benefit of using big data for enhancing security intelligence
is that huge swathes of business data can be stored in a secure and
tamper-proof manner that is useful for historical analysis in order to
learn from trends, for supporting forensic and legal investigations,
and for proving compliance with corporate governance and regulatory
requirements. This is extremely useful in looking for patterns across
large data sets that could indicate threats such as fraud being committed in a covert manner over long periods of time, or insider threats
from disgruntled or other employees whose behaviour is exposing the
organisation to risk. With such forensic capabilities, organisations will
be able to look for the root cause of security incidents; identifying, for
example, how a piece of malware moved through systems and which
systems were accessed, changed or potentially compromised over a
certain time period. This will allow the organisation to identify causes of
attacks in order that it can learn from such events and prevent similar
incidents from occurring in the future.
Big data analytics forensics capabilities are also useful in protecting
against the damage caused by so-called advanced persistent threats
(APTs) that aim not only to infiltrate a network, but to maintain a
presence on those networks over long periods of time, using techniques specifically aimed at avoiding detection. According to the SANS
Institute, more than half of Fortune 500 organisations have been targeted by such APTs7. The need for effective forensic analysis can be
seen in research conducted recently by Verizon Business, which found
that 85% of respondents have suffered data breaches that took weeks
or more to discover8.
The ability to collect, store and analyse massive amounts of log and event data from
multiple, disparate sources.
Keep that data highly compressed for costeffective storage and future analysis for
forensic purposes.
Query and analyse huge volumes of compressed data, often amounting to terabytes
of records, quickly and efficiently, in almost
real time.
Provide a rich set of analytical and reporting capabilities that converts information
regarding security events that have occurred into actionable intelligence for faster
remediation, improving the overall security
posture of the organisation.
For the system to work effectively, it must
also provide a front end analytics console that
enables security intelligence to be applied to
the data resulting from queries and analysis of
records from the event data warehouse. This
console provides the means through which
usersand not just those from IT, but general
business users who need access to complex
and ad hoc information reports for real time
status information and forensic analysis
interact with the system. This must provide the
means for users to directly access the back
end data for search and query purposes in an
easy manner, negating the need for users to
request reports be generated for them by IT
staff, which can be a serious bottleneck.
The console should also provide management
and administrative tools, including those for
constantly monitoring the data flows in real
time to look for abnormalities that could indicate suspicious behaviour. When such behaviour is uncovered, an alert should automatically
be generated and sent to relevant resources
to ensure speedy remediation can be taken.
Those alerts should also take into account
information such as that from vulnerability
The ability to run queries that can monitor and correlate events in real time so
that members of management and security teams can ensure that risks to the most
critical assets are minimised.
Providing the ability to take a more proactive stance on security through inclusion of
up-to-date threat information, rather than
just relying on reactive, signature-based
systems.
For a big data security and analytics system to be effective, all parts
of it need to be tightly integrated, built from the ground up to work together effectively by design, rather than separate technology systems
that require a high degree of integration. This requires that the system
be a combination of an event data warehouse, empowered by sophisticated analysis and security intelligence tools capable of analysing
voluminous and varied data sets; it needs to have been designed with
the needs of security in mind, supporting complex and ad hoc analysis
of time-stamped events to uncover threats and related patterns in real
time; and to support forensic investigations and compliance reporting
requirements.
Owing to the volumes of data that must be processed and analysed,
it must be hugely scalable and capable of handling an ever-growing
number and variety of data sets so that all event data generated by the
organisation can be included so that there are no gaps left in protection.
However, for performance purposes, it must be capable of compressing
data so that storage requirements are reduced, allowing the system to
scale more effectively.
There are a variety of technology solutions available, some of which
only incorporate some of the capabilities required for big data analysis
and most of which were not designed with integration in mind. This is
something that is problematic for most organisations as many lack the
resources in-house and many of the tools available have a wide variety
of configuration options for fine-tuning systems before they can be truly
useful. It is well recorded that many of the back end systems available
are complex to deploy, making integration an even harder task.
The integration challenges of systems that were designed in isolation
will become even more pressing as numerous surveys are reporting
that there is a shortage of resources with the skillsets required for big
data security. In a recent survey undertaken by AIIM, more than 50%
of respondents reported that big data analytics would be very useful,
but we dont have the skillsets.9 The survey also found that big data
skillsets are among the most sought after for respondents.
Given the lack of skilled resources, the expense of hiring consultants
and the integration challenges posed by cobbling together systems that
were designed for specific purposes, rather than as one homogenous
system, a far better choice is one unified system with common, centralised management tools that were not only designed for big data
analytics, but also for handling the complex nature of big data security
event information. A more general-purpose system built around big
data analytics, but not the specifics of big data security analytics, will
not only increase the implementation challenges, but will also require
heavy lifting by the security team, which negates the benefits of having
the data analysis available to all members of the organisation.
Summary
Many organisations are starting to realise
the benefits that harnessing big data brings.
In terms of big data analysis for operational
performance improvements, there are many
challenges to overcome. But big data security
analytics is even more challenging owing to
the complex nature of security information
and the specifics that are required for collecting, storing and processing security event
information so that its integrity is maintained.
This requires the use of an event-driven data
warehouse, specifically designed for compliance and discovery purposes, as opposed to a
traditional warehouse, which is better suited
for more basic analytics than search-based
environments. An event data warehouse
specifically designed for queries against
time-stamped data provides the only reliable
method of showing threat events and patterns
related to security.
In order to be able to effectively turn the information provided by the event data warehouse
into actionable intelligence that will guide the
organisation in improving its overall security
posture, tight integration needs to be natively
provided with advanced security intelligence
capabilities at the front end, provided through
a single console with advanced, centralised
management tools, and available to all users
in the organisation, not just security and IT
specialists.
Only through one integrated system, designed
so that all parts work together out of the box,
can organisations achieve the promised benefits of reduced cost and higher performance.
It must also be able to scale across multiple,
disparate and voluminous event data sets
and to be able to store massive data sets in
an efficient, cost-effective manner. Analytical
capabilities need to be integrated across all
parts of the system, with data collection and
analytics being uniform and intuitive for users,
without the need for proprietary tools that do
not effectively support integration across all
parts of the system.
References
1. http://www.emc.com/collateral/analystreports/idc-the-digital-universe-in-2020.
pdf
2. http://www2.sims.berkeley.edu/research/
projects/how-much-info-2003/
3. http://www.deloitte.com/view/en_GX/
global/industries/technology-mediatelecommunications/tmt-predictions-2012/
technology/70763e14447a4310VgnVCM100
0001a56f00aRCRD.htm
4. http://www.mckinsey.com/insights/mgi/
research/technology_and_innovation/
big_data_the_next_frontier_for_innovation
5. http://searchsecurity.techtarget.com/
news/2240157901/Gartner-Big-datasecurity-will-be-a-struggle-but-necessary
6. http://blog.varonis.com/big-data-security/
7. http://computer-forensics.sans.org/
blog/2012/06/02/the-apt-is-already-inyour-network-time-to-go-hunting-learnhow-in-new-training-course-sans-for508
8. http://www.verizonbusiness.com/
resources/reports/rp_data-breachinvestigations-report-2012_en_xg.pdf
9. http://www.aiim.org/Research-andPublications/Research/AIIM-White-Papers/
Career-Development
Further Information
Further information is available from
http://www.BloorResearch.com/update/2156
Fran Howarth
Senior Analyst - Security
Fran Howarth specialises in the field of security, primarily information security, but with a keen interest
in physical security and how the two are converging.
Frans other main areas of interest are new delivery models, such as cloud computing, information
governance, web, network and application security,
identity and access management, and encryption.
Fran focuses on the business needs for security technologies, looking at
the benefits they gain from their use and how organisations can defend
themselves against the threats that they face in an ever-changing landscape.
For more than 20 years, Fran has worked in an advisory capacity as an
analyst, consultant and writer. She writes regularly for a number of publications, including Silicon, Computer Weekly, Computer Reseller News,
IT-Analysis and Computing Magazine. Fran is also a regular contributor to
Security Management Practices of the Faulkner Information Services division of InfoToday.
2nd Floor,
145157 St John Street
LONDON,
EC1V 4PY, United Kingdom
Tel: +44 (0)207 043 9750
Fax: +44 (0)207 043 9748
Web: www.BloorResearch.com
email: info@BloorResearch.com