Technical

Workshops
Getting Started User
Training
Name
Title
Date

Agenda

Getting Started with Splunk

Deployment and Integration

Copyright

2011, Splunk Inc.

Listen to your data.

Getting Started With

Splunk

Platforms

Server Operating System

Unix (Linux, Solaris, FreeBSD)

Windows (XP, Vista, Win Server 2003/2008)

Max (OS10)
Storage 12-48% of raw data size.
Browsers

Firefox, IE6+, Safari 4

Versions

Community No cost, reduced features

Enterprise Full features

Copyright

2011, Splunk Inc.

Listen to your data.

Where to download

Software download

Documentation

http://www.splunk.com
http://www.splunk.com/base/Documentation

Whitepapers

Copyright

http://www.splunk.com/page/securelink/download/Splunk_Produc
t_Datasheet/

2011, Splunk Inc.

Listen to your data.

Install Splunk
www.splunk.com/down
load
32

or 64 bit?

Indexer

or Universal
Start Splunk

Forwarder?
WIN: \Program Files\Splunk\bin\splunk.exe start
Other: /opt/splunk/bin/splunk start

Splunk Home
WIN:

\Program Files\Splunk

Other:
Copyright

/opt/splunk (Applications/splunk)

2011, Splunk Inc.

Listen to your data.

Splunk Licenses
Free Download Limits Indexing to 500MB/day
Enterprise Trial License expires after 60 days
Reverts to Free License

Features Disabled in Free License

Multiple user accounts and role-based access controls

Distributed search
Forwarding to non-Splunk Instances
Deployment management
Scheduled saved searches and alerting
Summary indexing

Other License Types

Enterprise, Forwarder, Trial

Copyright

2011, Splunk Inc.

Listen to your data.

Splunk Web Basics

Browser Support

Firefox 2, 3.0.x, 3.5

Internet Explorer 6, 7, 8
Safari 3
Chrome 9

Default on install is http://localhost:8000

Index some data

Add data
Getting Started App
Install an App (Splunk for Windows, *NIX)

Copyright

2011, Splunk Inc.

Listen to your data.

Splunk Web Basics cont.

Splunk Apps

Splunk Home -> Find more apps

Apps create different contexts for your data out of sets of
views, dashboards, and configurations
You can create your own!

Search is an App

Summary will show everything you have indexed

Updated in real-time
Click on any source, sourcetype, or host to look at events

Copyright

2011, Splunk Inc.

Listen to your data.

Searching
Search > *
Select Time Range
Historical, custom, or real-time
Using the timeline
Click events and zoom in and out
Click and drag over events for a specific range

Copyright

2011, Splunk Inc.

10

Listen to your data.

Searching cont.

Search for any keyword

Search > error

Use Boolean expressions

Search > error OR failed NOT application
Spaces are implied AND, Operators need to be in caps
Search > Audit Failure = Audit AND Failure
Use quotes to search for a specific string
Search > Audit Failure
Use wildcards
Search > 46* OR 49*
Copyright

2011, Splunk Inc.

11

Listen to your data.

Search Assistant
Contextual Help

advanced typeahead

History

search
commands

Search Reference

short/long description
examples

Copyright

2011, Splunk Inc.

12

Listen to your data.

Deployment and
Integration

Splunk Has Four Primary Functions

Searching and Reporting (Search Head)
Indexing and Search Services (Indexer)
Local and Distributed Management (Deployment
Server)
Data Collection and Forwarding (Forwarder)

A Splunk install can be one or all

roles
Copyright

2011, Splunk Inc.

14

Listen to your data.

Getting Data Into Splunk

Agent and Agent-less Approach for Flexibility
Local File
Monitoring

syslog
TCP/UD
P

log files, config files

dumps and trace files

syslog compatible hosts

and network devices

Mounted File
Systems
\\hostname\mount

WMI
Event Logs
Performance

Unix, Linux and Windows

hosts

Copyright

Active
Directo
ry

shell scripts
custom parsers
cod batch loading

shel
l

perf

Event Logs
performance counters
registry monitoring
Active Directory
monitoring

virtual
host

Custom apps and scripted API

connections

Windows hosts

Agent-less Data
Input

2011, Splunk Inc.

Scripted
Inputs

Windows Inputs

Splunk Forwarder

15

Windows
hosts

Listen to your data.

Parsing
parsing pipeline actually consists of three pipelines:
parsing,
merging, and
typing,
which together handle the parsing function

Copyright

2011, Splunk Inc.

16

Listen to your data.

parsing
Extracting default fields for each event, - host,
source, and sourcetype.
character set encoding.
Identifying line termination using linebreaking rules.
Identifying timestamps or creating them if they don't
exist.
mask sensitive event data
configured to apply custom metadata to incoming
events.
Copyright

2011, Splunk Inc.

17

Listen to your data.

Indexing
Breaking all events into segments that can then be
searched upon.
Building the index data structures.
Writing the raw data and index files to disk, where
post-indexing compression occurs

Copyright

2011, Splunk Inc.

18

Listen to your data.

Indexing
Preconfigured indexes, including:
main: This is the default Splunk Enterprise index. All
processed data is stored here unless otherwise
specified.
_internal: Stores Splunk Enterprise internal logs and
processing metrics.
_audit: Contains events related to the file system
change monitor, auditing, and all user search history.

Copyright

2011, Splunk Inc.

19

Listen to your data.

Copyright

2011, Splunk Inc.

20

Listen to your data.

Understanding the Universal

Forwarder
Forward data without negatively impacting production
Monitor All
Supported
Inputs

performance.
Universal Regular
Universal Forwarder
Forwarde (Heavy)
Deployment
r
Forwarder
Logs
Messag
Configurations
Metrics

es

Routing,
Filtering,
Cloning

Splunk
Web

Python
Libraries

Event
Based
Routing

Scripted
Inputs

Copyright

2011, Splunk Inc.

Scripts

Central Deployment Management

Monitor files, changes and the system registry; capture metrics

and status.

21

Listen to your data.

Search
A search peer is an indexer that services requests from
search heads in a distributed search deployment.
Search peers are also sometimes referred to as indexer nodes.
A search head is a Splunk instance configured to distribute
searches to indexers, or search peers.
Search heads can be either dedicated or not, depending on
whether they also perform indexing.
Dedicated search heads don't have any indexes of their own
(other than the usual internal indexes). Instead, they consolidate
results that originate from remote search peers.

Copyright

2011, Splunk Inc.

22

Listen to your data.

Functions at a glance
Functions

Indexer

Indexing
Web
Direct search
Forward to
indexer
Deploy
configurations

Copyright

2011, Splunk Inc.

Search head

Forwarder

Deployment
server

x
x
x
x
x

23

Listen to your data.

Horizontal Scaling
Load balanced search and indexing for massive, linear
scale out.
Distributed
Search

Forwarder
Auto Load
Balancing

Copyright

2011, Splunk Inc.

24

Listen to your data.

Multiple Datacenters

Index and store locally. Distribute searches to datacenters, networks

& geographies.
Headquarter
s

Distributed
Search

London

Copyright

2011, Splunk Inc.

Hong Kong

Tokyo

25

New York

Listen to your data.

Data Redundancy
Clone data to multiple index servers to eliminate a single point of
failure.

Active

Hot Standby

Forwarding to DR site
Data
Cloning

Copyright

2011, Splunk Inc.

26

Listen to your data.

High Availability

Combine auto load balancing and cloning for HA at every Splunk

tier.
Shared Storage
Distributed
Search
Clone Group 1 : Complete Dataset

Distributed
Search
Clone Group 2 : Complete
Dataset

Data Cloning &

Auto Load
Balancing
Copyright

2011, Splunk Inc.

27

Listen to your data.

Send Data to Other Systems

Route raw data in real time or send alerts based on searches.
Service
Desk
Event
Console
SIEM

Copyright

2011, Splunk Inc.

28

Listen to your data.

Integrate External Data

Extend search with lookups to external data sources.
LDAP,
AD

Watch
Lists

CMDB

CRM/E
RP

Correlate IP addresses with locations, accounts

with regions

Copyright

2011, Splunk Inc.

29

Listen to your data.

Integrate Users and Roles

Integrate authentication with LDAP and Active Directory.
LDAP, AD
Users and Groups

Problem Investigation

Splunk Flexible Roles

Capabilities & Filters

Manage
Indexes

Problem Investigation

Share
Search Save
es
Searche
Problem
Investigation
s
Manage
Users
NOT
tag=PCI
App=ER
P

Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter.
Copyright

2011, Splunk Inc.

30

Listen to your data.

Centralized Licensing Management

Groups, Stacks, and Pools for Enterprise Deployments

Problem Investigation

Copyright

2011, Splunk Inc.

31

Listen to your data.

Deployment Monitoring
Keep Tabs On Your Splunk Enterprise Deployment

Licenses
Copyright

2011, Splunk Inc.

Sourcetypes

Indexers
32

Forwarders
Listen to your data.

Scalability Use
CASE

Copyright

2011, Splunk Inc.

Listen to your data.

Scaling Splunk 101

Quick Overview of Scaling Splunk with Commodity Hardware
Erik Swan
Oct, 09

** Slides intentionally ugly, no designers were harmed during construction

Copyright

2011, Splunk Inc.

Listen to your data.

Single Server Install

Commodity Architecture
Data from Splunk
Forwarders, Syslog, Files,
etc.

Splunk

one)

(all in

Users

Copyright

2011, Splunk Inc.

Simplest Splunk install is a

single server that functions as
both indexer and search head.
A single box can easily index
100-200G per day, BUT for
fast searching its best to use
more than one box.

Listen to your data.

Improving Search and Indexing Performance

Splunk scales search and indexing performance
horizontally by adding more indexers and in some
cases scaling out a search tier.
By spreading the incoming load across more
indexers you index faster.
Perhaps more importantly, by spreading the indexed
data across more indexers your search performance
improves linearly as well.
Consider that every doubling of hardware will
double your index and search performance and
dont be shy of adding 10s of servers.
Copyright

RULE
#1
2011,
Splunk
Inc.

If your searches are slow, add another

Listen to your data.

Adding a Search Head

Data from Splunk
Forwarders, Syslog, Files,
etc.

Spunk
Indexer

By splitting out a Search Head,

search performance is improved
and load is taken off the indexer for
faster indexing.
Best to add sooner than later.

Splunk Search
Head

Users

Copyright

2011, Splunk Inc.

Best for volumes between 5-100G

p/day
1 Indexer
1 Search Head

Listen to your data.

Adding a second Indexer

Data from Splunk Forwarders, Syslog, Files,
etc.

Spunk
Indexer

Spunk
Indexer

Splunk Search
Head

Users

Copyright

2011, Splunk Inc.

As volume goes up beyond 100G OR

you want to improve search
performance its best to add a second
Indexer.
**Remember adding indexers improves
search performance linearly as well.
Best for volumes 20-200G p/day
2 Indexers
1 Search Head

Listen to your data.

Adding additional Indexers

TBs/day from Splunk Forwarders and Syslog

Spunk
Indexer

Spunk
Indexer

(n) Indexers

Splunk Search
Head

Users

Copyright

2011, Splunk Inc.

Spunk
Indexer

For every new ~100G, or

again to improve search
performance add another
indexer.
RULE #1: If searches are
slow, add an another
indexer.
For volumes from 200G-1T
p/day

Listen to your data.

Adding additional Indexers

TBs/day from Splunk Forwarders and Syslog

Assume 100G p/day:

: Log archival and some periodic

SpunkUse Case Spunk
Spunk
Indexer
Indexer
Indexer
troubleshooting
(n) Indexers

For every new ~100G, or

again to improve search
performance add another
indexer.

1 Commodity Server

RULE #1: If searches are

slow, add an another
Search
Use Case #2Splunk
: Archival,
troubleshooting and summary
indexer.
Head
reporting
1 Index Server, 1 Search Server
For volumes from 200G-1T
p/day
Use Case #3: Users
Archival, Trouble Shooting, and Reporting
2 Index Servers, 1 Search Server
Use Case #4: Many ( >2 ) users doing constant use
3+ Index Servers, 1 Search Server
Copyright

2011, Splunk Inc.

Listen to your data.

Adding additional Search Heads

TBs/day from Splunk Forwarders and Syslog

Spunk
Indexer

Spunk
Indexer

Splunk Search
Head

(n) Indexers

(n) Search Heads

1~ 4T each p/day

Load Bal.

Spunk
Indexer

Splunk Search
Head

Adding more Search Heads is a

convenient way to improve
search performance
Add an additional Search
Heads when:
1. It makes sense to partition
users.
2. Too offload summary or
scheduled searches.

Users

Copyright

2011, Splunk Inc.

Listen to your data.

Adding additional Search Heads

TBs/day from Splunk Forwarders and Syslog

Assuming a load of 1T p/day:

Spunk
Indexer

Spunk
Indexer

Spunk
Indexer

For every new ~TB p/day, add

another search head.

(n) Indexers and some periodic

Use Case #1: Log archival
For volumes > 2T p/day
troubleshooting
4 Index Servers, 1 Search Server
(n) Indexers each <100G p/day

Splunk Search
Head

(n) Search Heads

#2:
Archival,
1~ 4T
each p/day

Splunk Search
Head

Use Case
trouble shooting and
summary reporting
8+ Index Server,
Load Bal. 1 Search Server

(m) Search Heads for every ~1T

p/day
some

Use Case #3: Archival, Trouble Shooting, and

Reporting Users
16+ Index Servers, 1 Search Server
Use Case #4: Many ( >2 ) users doing constant use
Copyright 2011,
Splunk
Inc. Servers, 1 Search Server
20+
Index

Listen to your data.

Long term storage, add a SAN

TBs/day from Splunk Forwarders and Syslog

Spunk
Indexer

Spunk
Indexer

(n) Indexers

If wanting to keep more than

can be kept on local indexer
disk, splunk can be configured
to use SAN or other storage
device.

Tier 1
SAN
Splunk Search
Head

Spunk
Indexer

Long term storage can not be

kept on local commodity IO.

Splunk Search
Head

Best for keeping >30 day

multi year data.

Load Bal.

Users
Copyright

2011, Splunk Inc.

Listen to your data.

Multi-datacenter or deployment

If you have multiple data centers, it is often best to leave the data local
and use distributed search between two deployments.
If you have data that naturally partitions such that users would rarely
search across the data, partitioning entire deployments can help.
Obviously for DR as well.

Copyright

2011, Splunk Inc.

Listen to your data.

Additional Scaling Topics

Summary Indexing If your searches are slow consider using
summary indexing:
1. video - http://www.splunk.com/view/SP-CAAACZW
2. docs http://www.splunk.com/base/Documentation/4.0.5/User/Use
SummaryIndexingForIncreasedReportingEfficiency
Routing High Volume data to Separate Index If you are
searching or reporting on a source that is dwarfed by the
volume of another source, you can partition data such that the
high volume source is in its own index:
3. docs http://www.splunk.com/base/Documentation/latest/Admin/S
etupmultipleindexes#Why_have_multiple_indexes.3F

Copyright

2011, Splunk Inc.

Listen to your data.

Support and Community

Support Through the Splunk

Community
Splunkbase

Browse and share

Apps from Splunk,
Partners and the
Community
splunkbase.splunk.c
Copyright 2011,
omSplunk Inc.

Community-driven
knowledge
exchange and Q&A
answers.splunk.com
47

5 tracks, more than

40 sessions, the
smartest Splunk
users together
www.splunk.com/goto/conf
erence
Listen to your data.

Where to Go for Help

Documentation

http://www.splunk.com/base/Documentation

Technical Support

http://www.splunk.com/support

Videos

http://www.splunk.com/videos

Education

http://www.splunk.com/goto/education

Professional Services

Copyright

2011, Splunk Inc.

48

Listen to your data.