TO SIEM OR NOT TO SIEM

Suggestions for Getting a Handle on Event Logs

Ray Sprong
January 22, 2011
Now that is the question, isn’t it? Whether it is nobler to deploy a SIEM now or focus on your log
consolidation and reporting. What the heck is SIEM anyway these days? The entire marketplace is
confused. Vendors and analysis groups are not making it any easier either. I hope this paper will clarify
some of the misconceptions and confusion that is rampant.

First let’s deal with the alphabet soup. SEM (Security Event Management), SIM (Security Information
Management) and SIEM (Security Information Event Management) are all pronounced the same and
that used to be okay, since they used to mean essentially the same thing. The acronyms came about by
vendors attempting to differentiate their product from the competition, but essentially they did the
same thing. For the ease of writing and simplification, we will use the acronym SIEM.

SIEM systems came about shortly after the introduction of Intrusion Detection Systems (IDS). Intrusion
Detection Systems, although great in concept, were getting a black eye quick because of the
proliferation of false positives that the product produced. It is the old ‘Boy who cried wolf’ scenario.
Some industrious soul introduced the first SIEM by taking the events generated by an IDS and comparing
those events to the events received from the firewalls. IF we get THIS event from the IDS and we see
THAT event from the firewall then you have a PROBLEM. More simply put: IF THIS + THAT = PROBLEM.
This is simply called ‘Correlation’. Correlation is the ability to compare different events for a common
purpose. It is the SIEM system’s ability to correlate event information that made it powerful and
popular. I am sure you have heard the elderly say “Life was simpler back then.” and it was. If you
purchased an IDS, then you also needed to get some type of SIEM in place or you would spend a lot of
time dealing with problems that didn’t exist (the definition of “False Positive”).

Vendors who sold some type of SIEM were having a hay day and a lot of them popped up, each with
differentiating features and therefore the acronym soup discussed previously. The important thing to
remember with SIEM systems is their focus on the ‘S’ of SIEM – Security. SIEM systems were designed
to deal with ‘Security Events’. The systems were (and still are) designed around filtering out non-
security events to reduce the volume associated with global event logging. For those of you who know,
network devices (firewalls, routers, and switches), servers (hardware and operating systems),
applications, etc. all create logs – and a lot of them. SIEM systems had to reduce the sheer volume in
order to be able
100%

<100%

This document is considered proprietary information of ITAT Partners. Such proprietary information may not be used, reproduced, or disclosed to any other parties
for any purpose without the expressed written permission of ITAT Partners.
to process the ‘events that count’. That was okay, because the buzz word of the day was ‘Security’.
Security was where customers were spending their lion’s share of their budget dollars and that was
definitely the place to be.

A few companies, with foresight, were attempting to make their IT operations aligned with quality
frameworks such as ITIL, COBIT, ISO, etc. Aligning your IT operation with a chosen framework is
definitely the way to go and, in fact, if more companies had done it, the need for compliance mandates
would never have materialized, but I digress. Part of any quality framework is the requirement to
review your logs – not just your security event logs, but all your logs. Reviewing your event logs on each
system is an arduous task at best. The frameworks ‘sort of’ defined what you should be looking for, but
every event is different, and there are a whole lot of them. Let’s just say, it was easier to find the needle
in the haystack then it was finding the event log that you needed to see. The bigger the network, the
more problematic it becomes. So these companies started looking around for systems that would
consolidate all of those the logs associated with operations, systems, and security into a central location
and provide some alerting and reporting on what was important.

Thus the Log Management and Intelligence (LMI) space was created and a whole new set of vendors, led
by LogLogic, Inc. came to be. This was the beginning of the “Confusing Times”. Even analysis groups like
Gartner didn’t know what to do with these upstarts. They were definitely not SIEM systems because
they lacked the correlation capabilities, but they were definitely a force to reckon with. The ability to
really provide correlation of all of the logs required too much processing power to deal with on a single
management system. This is where the fallacy of analysis systems like Gartner’s Magic Quadrant comes
in to play. Gartner does a phenomenal job of rating a vendor’s capability, but the assumption is that the
vendor’s are all doing the same thing. A company that does not fare well in the Magic Quadrant could
be a great solution, but is just being looked at in the wrong way. This is what happened to Log
Management companies – the only place to evaluate them was in the SIEM quadrant and thus, because
of their lack of SIEM functionality, they were rated poorer than what they really were.

Through lobbying and other tactics, these Log Management companies finally got their own Quadrant,
entitled Log Management & Intelligence (LMI), and decisions became clearer. Those that were
interested in managing their logs purchased an LMI solution. Those that were interested in just their
security events purchased a SIEM. Those that were interested in both, purchased – yes you got it –
both.

Along about the same time as all of this was going on, Sarbanes Oxley (SOX) came on the scene. The
SOX requirement was followed shortly by the Payment Card Industry Data Security Standard (PCI-DSS or
PCI for short) mandate. All of a sudden, the buzz word shifted from ‘security’ to ‘compliance’ and so did
company’s spending budgets. LMI solution vendors were very well placed for this shift because all of
the compliance requirements, whether SOX (publicly traded companies), PCI (companies dealing with
credit cards), NERC (energy companies), FISMA (banking industry), etc. are all based on one of the
frameworks that define quality in IT operations, which required review of all event logs, not just security
logs. Because spending shifted, so did the sales pitch of SIEM vendors and now we enter into the next
phase of confusion. SIEM vendors claimed the ability of providing compliance reports and they did –
centered around the security events they collected – remember SIEM, by definition, filter events at the
source, so are not looking at all of a IT operation’s logs. LMI vendors were able to create compliance
suites around the various compliance requirements and really make an impact on a company’s
compliance direction and needs. So, you guessed it, more LMI solutions were purchased in lieu of SIEM

This document is considered proprietary information of ITAT Partners. Such proprietary information may not be used, reproduced, or disclosed to any other parties
for any purpose without the expressed written permission of ITAT Partners.
systems. Remember, a company may only have so much to spend and so they had to make a choice.
The fortunate companies acquired both.

SIEM providers now began building into their solutions some type of LMI capabilities. They had to in
order to try to beef up falling sales numbers, but remember, their basic architecture was not designed to
handle the volume of logs represented by the collection of ‘all’, so, at best, their LMI capability was
limited. What they did do, however, was confuse the marketplace even further because the lines of
demarcation were not as clear. To add to the mix, LMI vendors began to offer SIEM capabilities, but this
was mostly done through acquisition. And now, to even confuse the options even more, IDS solutions
(remember – they started this whole thing) have gotten smarter and most IDS companies now offer
Intrusion Prevention Solutions (IPS) which encompasses some SIEM like functions with the ability to
automatically remediate the problems. As I had one customer put it, automatic remediation is a little
like giving a monkey are really big hammer and saying “have at it” on your network. Suffice it to say,
that even with today’s technology, I am still not a big fan of automatic remediation, but that is a subject
for another discussion.

That pretty much brings us up to today’s mix. The Gartner Group, from what I understand, has dropped
the LMI quadrant and is once again lumping everyone into the same grouping, so be careful that you
know what you are looking for when you look at their quadrants.

So here is the question that I have been asked many times over the years – “Which should I deploy first
– SIEM or LMI?”. The obvious answer is in the form of a question – “Where is your pain?” In most
cases, I strongly recommend that a company deploy a solid LMI solution first. Get a handle on your logs.
Establish a baseline of performance of everything from who is accessing your systems, failed logins,
allowed ports, and, oh yes, your log volume in general. So much can be learned about the health of your
network, simply by looking at your log volume generated, by each source, from day to day. Logs are
predictive in nature, which is to say, the volume should be relatively the same from day to day unless
something has changed or there is a problem. I have personally saved companies small fortunes in
needless expenditures simply by looking at volume deviations and helping customers understand the
causes of those deviations. You also need to not only see who has logged in, but who has attempted to
login historically. Early on there was a customer who through log forensics was able to identify an ex-
employee who had logged in after his employment termination. This forensic capability enabled the
customer to recover stolen electronic assets prior to any of it becoming public information.

This brings up the other reason for deploying a solid LMI solution first - the need for long-term log
retention. Best practices dictate that you retain your logs for a minimum of one year. It is not sufficient
to just store your logs for that period of time in any fashion. A log, by definition, is the unadulterated
truth of what is going on and what has gone on in your network. It is considered as evidence and, as
such, it needs to be treated like evidence. Therefore, there needs to be a chain of evidence established
and the logs need to be retained in such a way as to insure that they are not tampered with. If ever
needed for judicial purposes, the event logs need to be verifiably unaltered.

SIEM vendors will argue that you should be catching these critical events when they happen, which you
can, if you provide the care and feeding that a SIEM solution requires and your crystal ball is working so
that you know every question that you could possibly ask.

This document is considered proprietary information of ITAT Partners. Such proprietary information may not be used, reproduced, or disclosed to any other parties
for any purpose without the expressed written permission of ITAT Partners.
This brings up yet another factor to consider. LMI is easy to implement relative to a SIEM and requires
very little attention to keep it functioning and providing value. SIEM solutions, on the other hand, can
be difficult to deploy requiring time and professional services, and, once deployed, can require constant
attention to keep accurate alerting enforced. Similar to network management solutions, if you don’t
provide the attention to a SIEM solution, over time, the number of false alerts that it generates, will
make operations personnel ignore it and your investment will have been wasted. Deployment of a SIEM
solution can be made less difficult and the subsequent daily maintenance easier if you have a solid
baseline of your logging environment and know what you are looking for. This intelligence can only
come about through a previous deployed log management solution. LMI, on the other hand, does not
require daily feeding to make sure it remains valuable. It will sit there and just collect logs. The better
LMI solutions will provide automatic reporting and alerting if configured, but for the most part the
solution just keeps on chugging along. One pitfall that can bite you though is a result of this fact. With
any LMI, because you don’t have to log into it to adjust it constantly, there is the exposure that one of
the log sources that you think you are collecting logs from will stop sending logs. This can occur for
three reasons: 1 – The source itself has died, 2- Something in the path has changed which is not
permitting the logs through, or 3 – Someone or something has turned logging off, which very likely could
be with malicious intent. The better logging systems will monitor this and alert you to the instance that
it has stopped receiving logs from a specific source.

When discussing the options between LMI and SIEM, the 80/20 rule comes into play. 80% of the log
functionality that an IT organization requires can be accomplished with a viable, strong LMI solution and
typically for 20% of the cost. I strongly recommend that companies who, today, don’t have any real idea
of their log volume or content, deploy an LMI solution and spend time ‘cleaning’ up their environment
by getting rid of the low hanging fruit associated with access control, firewall rules and control, and
network operations. Then, do a gap analysis between what you have with LMI and what you still need
in the form of log management operations. If the 20% (at most) of your unsatisfied requirements are
worth the additional 80% investment, then deploy a SIEM in your environment. SIEM vendors will tell
you that relative to LMI, they are not 80% more expensive, but what they won’t tell you is the cost
associated with professional services that will be required to get it operational and the on-going full
time equivalent (FTE) resources that will be needed to keep it going. Those are true costs to an
organization that, if not factored in, will not be there to spend when it is needed and the solution will
fail.

Please do not take from this paper that I do not like or recommend SIEM solutions for your
environment. My position is quite the contrary. I strongly feel that a properly deployed SIEM in an
enterprise network can bring significant value and peace of mind and is worth every penny you spend
on it over the long term. It is just that deployment of a SIEM prematurely on your network is likened to
purchasing a $200,000 yacht with no boating experience. It can be done, but not without headaches, a
huge learning curve, and more investment than what you originally bargained for. Don’t forget – it is

This document is considered proprietary information of ITAT Partners. Such proprietary information may not be used, reproduced, or disclosed to any other parties
for any purpose without the expressed written permission of ITAT Partners.
also possible to sink. LMI is much easier to install and maintain and provides you the ability to learn
your environment and more importantly learn what your environment needs.

So now let’s discuss the hybrids. You know - those solutions that originally started out as SIEM
solutions, but now, “oh by the way”, do log management also. Or those solutions that originally started
out as LMI platforms, but now, “oh by the way”, do SIEM functionality. Be careful! Do not assume that
just because a solution is good at one that they will be good at the other. In most cases, I have found
that the original solution may be great, but the addition leaves a lot to be desired. It appears almost like
it was added as an afterthought in order to continue to be competitive. Furthermore, don’t be
sidetracked by feature rich add-ons that have nothing to do with your goal. There are always nice to
have enrichments to products, but they probably shouldn’t be considered requirements. The ability to
handle Flow Control streams is the example that comes to mind. There are excellent flow control
analyzers that rock. Flow control analysis is a totally different science. There is an old adage: “Jack of all
trades, master of none”. Be careful you don’t load up all of your nice to have features and wind up with
a solution that is a master of none.

Okay, so where are you now? In summary, I suggest that you take the approach to getting a handle on
log management slowly and methodically. Follow the steps in the figure below to help insure that you
are using your company’s funds intelligently.

1. 2.
Understand Identify Log
Desired Goal Sources

3.
Deploy
Consolidated Log
Collection (LMI)

4.
Baseline Log
Environment

5. 6.
Perform Gap Deploy SIEM that
Analysis best fits the needs

1. Establish a goal for where you want to wind up. If compliance is your driver, understand
what you need to get out of your solution before you start looking. If it is a framework that
you are managing toward, understand what the framework is suggesting with respect to
your logs. Remember: neither compliance nor frameworks require correlation. You can
adjust your goal as you go through the process, but make those adjustments aligned with
your original intention.
2. Identifying your log sources sounds easy and it very well could be. If compliance is your
driver, then you may not need to collect logs from every device. For example, PCI states
that you only have to review your logs from those devices that process, stores, or transport

This document is considered proprietary information of ITAT Partners. Such proprietary information may not be used, reproduced, or disclosed to any other parties
for any purpose without the expressed written permission of ITAT Partners.
 credit card information. The best advice is to identify your most critical assets and start
there. A word of caution, though: inevitably what you start with and why you start with it
will change. Every implementation of log management always grows because of the value
that is obtained from the knowledge built into log events.
3. Consolidate all of your logs from the sources identified in step 2. Time has already been
spent discussing this step, but this is a critical step that is often skipped due to budgetary
restrictions or confusion in the marketplace.
4. There is so much that is going on in your network that you are unaware of. I have yet to
implement a log management solution on a customer’s network and not have the customer
identify ‘issues’ that they weren’t aware of. It is that simple. There are low hanging fruit
like an automated login that someone set up years before that doesn’t work any longer and
it strings out failed logons in the thousands. There are also some things that require work
like the identification of ports that are active inside the network – understanding what they
are, who is using them, and deciding if it should be continued to be allowed. It is important
in this step, not just to understand your event logging, but clean up the items that you
discover. Your network and your users will thank you for it.
5. Now that you have done all of your homework, this step is easy. What did you identify in
Step 1 that is not yet satisfied? Is what you identified in Step 1 still that important to you?
If it is, seek a specific solution to those needs. One key advantage to seek out, if you move
to Step 6, is to try to find a solution in Step 3 that will allow for log forwarding and a solution
in Step 6 that will accept those logs. A number of syslog servers will forward logs, but
overlay the syslog server’s IP address on each event it forwards. This makes every event
look like it was generated by the syslog server instead of the source device itself. That is very
bad.
6. If you have done all of the above steps, this implementation will be very easy. You will know
specifically what you want to correlate and what your output should look like. Any SIEM
vendor will appreciate this because it allows them to focus their professional services
specifically to meet your needs. Of course, your needs will change and evolve, but you will
have the foundation to adjust the tools as it happens. So, make sure that you acquire a
system that YOU can manage. Dependency upon an outside party to create rules, reports,
alerts, or anything else creates a situation that you will not be able to control over the long-
term.

About the author: Ray Sprong is the president of ITAT Partners, an independent reseller of network
tools and test equipment that enhance the way IT staffs work toward compliance, security, and daily
operational efficiencies. Ray has been involved in security event and log management sales and
education since 2002. His career has included international and domestic field and center operations in
satellite and telecommunications where his focus was on operational efficiencies and streamlining
methodologies. Ray can be contacted at 630-789-1100 or at rsprong@itatinc.com.

This document is considered proprietary information of ITAT Partners. Such proprietary information may not be used, reproduced, or disclosed to any other parties
for any purpose without the expressed written permission of ITAT Partners.