Documente Academic
Documente Profesional
Documente Cultură
1
Forwarding Data
Generated: 11/23/2015 7:49 pm
Table of Contents
Introduction to forwarding..................................................................................1
About forwarding and receiving...................................................................1
Types of forwarders....................................................................................2
The universal forwarder..............................................................................6
Plan your deployment.........................................................................................9
System requirements..................................................................................9
Forwarder deployment topologies.............................................................10
Compatibility between forwarders and indexers.......................................15
Deploy a universal forwarder............................................................................16
Universal forwarder deployment overview................................................16
Enable a receiver......................................................................................18
Install the universal forwarder software.....................................................21
Consolidate data from multiple machines.................................................21
Migrate from a light forwarder...................................................................23
Install the universal forwarder software..........................................................25
Deploy a Windows universal forwarder via the installer GUI....................25
Deploy a Windows universal forwarder via the command line..................36
Remotely deploy a Windows universal forwarder with a static
configuration.............................................................................................50
Deploy a *nix universal forwarder manually..............................................52
Remotely deploy a *nix universal forwarder with a static configuration....57
Migrate a Windows light forwarder............................................................64
Migrate a *nix light forwarder....................................................................66
Configure the universal forwarder...................................................................68
Configure the universal forwarder.............................................................68
Configure forwarders with outputs.conf.....................................................70
Configure data collection on forwarders with inputs.conf..........................78
Supported CLI commands........................................................................80
Deploy heavy and light forwarders..................................................................82
Enable forwarding on a Splunk Enterprise instance.................................82
Enable a receiver......................................................................................83
Deploy a heavy forwarder.........................................................................86
Deploy a light forwarder............................................................................89
Heavy and light forwarder capabilities......................................................92
i
Table of Contents
Upgrade forwarders...........................................................................................95
Upgrade the Windows universal forwarder...............................................95
Upgrade the universal forwarder for *nix systems....................................98
Upgrade heavy and light forwarders.......................................................101
Perform advanced configuration....................................................................102
Set up load balancing..............................................................................102
Configure a forwarder to use a SOCKS proxy........................................106
Configure an intermediate forwarder.......................................................110
Protect against loss of in-flight data........................................................113
Route and filter data................................................................................119
Make a universal forwarder part of a system image...............................134
Forward data to third-party systems........................................................137
Configure forwarders with outputs.conf...................................................142
Troubleshoot forwarding................................................................................151
Troubleshoot forwarder/receiver connection...........................................151
Heavy and light forwarders.............................................................................154
Heavy and light forwarder capabilities....................................................154
ii
Introduction to forwarding
About forwarding and receiving
You can forward data from one Splunk Enterprise instance to another Splunk
Enterprise instance or even to a non-Splunk system. The Splunk Enterprise
instance that performs the forwarding is typically a smaller footprint version of
Splunk Enterprise, called a forwarder.
A Splunk Enterprise instance that receives data from one or more forwarders is
called a receiver. The receiver is usually a Splunk Enterprise indexer, but can
also be another forwarder.
Forwarders represent a much more robust solution for data forwarding than raw
network feeds, with their capabilities for:
Tagging of metadata (source, source type, and host)
Configurable buffering
1
Data compression
SSL security
Use of any available network ports
The forwarding and receiving capability makes possible all sorts of interesting
Splunk Enterprise topologies to handle functions like data consolidation, load
balancing, and data routing.
Types of forwarders
There are three types of forwarders:
The universal forwarder is a streamlined, dedicated version of Splunk
Enterprise that contains only the essential components needed to forward
data to receivers.
A heavy forwarder is a full Splunk Enterprise instance, with some
features disabled to achieve a smaller footprint.
A light forwarder is also a full Splunk Enterprise instance, with most
features disabled to achieve as small a footprint as possible. The light
forwarder has been deprecated as of Splunk Enterprise version 6.0. The
universal forwarder supersedes the light forwarder for nearly all purposes
and represents the best tool for forwarding data to indexers.
For a list of all deprecated features, see "Deprecated features" in the Release
Notes.
One key advantage of the heavy forwarder is that it can index data locally, as
well as forward data to another Splunk Enterprise instance. You must activate
this feature. See "Configure forwarders with outputs.conf" in this manual for
details.
A light forwarder has a smaller footprint with much more limited functionality. It
forwards only unparsed data. Starting with version 4.2, it has been superseded
by the universal forwarder, which provides very similar functionality in a smaller
footprint. The light forwarder continues to be available mainly to meet legacy
needs. We recommend that you always use the universal forwarder to forward
unparsed data. When you install a universal forwarder, the installer lets you
migrate checkpoint settings from any (version 4.0 or greater) light forwarder that
resides on the same machine. See "The universal forwarder" for a more detailed
comparison of the universal and light forwarders.
For detailed information on the capabilities of heavy and light forwarders, see
"Heavy and light forwarder capabilities."
To learn how to enable and deploy a heavy or light forwarder, see "Deploy a
heavy or light forwarder."
Forwarder comparison
This table summarizes the similarities and differences among the three types of
forwarders:
Features and
capabilities
Universal
forwarder
Light forwarder
Type of Splunk
Dedicated
Enterprise instance executable
Heavy forwarder
Full Splunk
Enterprise, with
most features
disabled
Full Splunk
Enterprise, with
some features
disabled
Footprint (memory,
CPU load)
Smallest
Small
Medium-to-large
(depending on
enabled features)
Bundles Python?
No
Yes
Yes
Handles data
inputs?
All types
All types
Forwards to Splunk
Yes
Enterprise?
Yes
Yes
Forwards to 3rd
party systems?
Yes
Yes
Yes
Serves as
intermediate
forwarder?
Yes
Yes
Yes
Indexer
acknowledgment
(guaranteed
delivery)?
Optional
Optional (version
4.2+)
Optional (version
4.2+)
Load balancing?
Yes
Yes
Yes
Data cloning?
Yes
Yes
Yes
Per-event filtering?
No
No
Yes
Event routing?
No
No
Yes
Event parsing?
No
No
Yes
Optional, by
setting
Local indexing?
No
No
indexAndForward
attribute in
outputs.conf
Searching/alerting? No
No
Optional
Splunk Web?
No
No
Optional
For detailed information on specific capabilities, see the rest of this topic, as well
as the other forwarding topics in the manual.
With raw data, the forwarder sends the data stream as raw TCP. it does not
convert the data into the Splunk communications format. The forwarder collects
the data and forwards it on. This is particularly useful for sending data to a
non-Splunk system.
With unparsed data, a universal forwarder performs minimal processing. It does
not examine the data stream, but it does tag the stream with metadata to identify
source, source type, and host. It also divides the data stream into 64-kilobyte
blocks and performs some rudimentary timestamping on the stream, for use by
the receiving indexer in case the events themselves have no discernible
timestamps. The universal forwarder does not identify, examine, or tag individual
events.
With parsed data, a heavy forwarder breaks the data into individual events,
which it tags and then forwards to a Splunk Enterprise indexer. It can also
examine the events. Because the data has been parsed, the forwarder can
perform conditional routing based on event data, such as field values.
The parsed and unparsed formats are both referred to as cooked data, to
distinguish them from raw data. By default, forwarders send cooked data in
the universal forwarder's case, unparsed data, and in the heavy forwarder's case,
parsed data. To send raw data instead, set the sendCookedData=false
attribute/value pair in outputs.conf.
Read on!
For information on deploying the universal forwarder, see the topics that directly
follow this one.
For information on third-party Windows binaries that the Windows version of the
Splunk Enterprise universal forwarder ships with, read "Information on Windows
third-party binaries distributed with Splunk Enterprise" in the Installation Manual.
For information about running the universal forwarder in Windows Safe Mode,
read "Splunk Enterprise Architecture and Processes" in the Installation Manual.
Licensing requirements
The universal forwarder ships with a pre-installed license. See "Types of Splunk
Enterprise licenses" in the Admin manual for details.
Other requirements
Sun SPARC systems
If you plan to install a universal forwarder on a Sun SPARC system that runs
Solaris, confirm that you have patch level SUNW_1.22.7 or later of the C library
(libc.so.1). If you do not, the universal forwarder cannot run because it needs
this version of the library.
User rights
You must have admin or equivalent rights on the machine where you're installing
the universal forwarder.
learn more about forwarders and clusters, read "Use forwarders to get your data"
in the Managing Indexers and Clusters of Indexers manual.
Data consolidation
Data consolidation is one of the most common topologies, with multiple
forwarders sending data to a single Splunk Enterprise instance. The scenario
typically involves universal forwarders forwarding unparsed data from
workstations or production non-Splunk servers to a central Splunk Enterprise
instance for consolidation and indexing. With their lighter footprint, universal
forwarders have minimal impact on the performance of the systems they reside
on. In other scenarios, heavy forwarders can send parsed data to a central
Splunk Enterprise indexer.
Here, three universal forwarders are sending data to a single indexer:
10
For more information on data consolidation, read "Consolidate data from multiple
machines".
Load balancing
Load balancing simplifies the process of distributing data across several
indexers to handle considerations such as high data volume, horizontal scaling
for enhanced search performance, and fault tolerance. In load balancing, the
forwarder routes data sequentially to different indexers at specified intervals.
Forwarders perform automatic load balancing, in which the forwarder switches
receivers at set time intervals. If parsing is turned on (for a heavy forwarder), the
switching will occur at event boundaries.
In this diagram, three universal forwarders are each performing load balancing
between two indexers:
11
12
For more information on routing and filtering, read "Route and filter data".
13
To learn more about forwarders and indexer clusters, read "Use forwarders to get
your data" in the Managing Indexers and Clusters of Indexers manual. To learn
more about indexer clusters in general, read "About indexer clusters and index
replication".
Intermediate forwarding
To handle some advanced use cases, you might want to insert an intermediate
forwarder between a group of forwarders and the indexer. In this type of
scenario, the originating forwarders send data to a consolidating forwarder, which
then forwards the data on to an indexer, usually after indexing it locally.
Typical use cases are situations where you need an intermediate index, either for
14
15
Steps to deployment
The actual procedure varies depending on the type of deployment, but these are
the typical steps:
1. Plan your deployment.
2. Download the universal forwarder from
http://www.splunk.com/download/universalforwarder
16
Enable a receiver
To enable forwarding and receiving, you configure both a receiver and a
forwarder. The receiver is the Splunk Enterprise instance receiving the data; the
forwarder sends data to the receiver.
Depending on your needs (for example to enable load balancing), you might
have multiple receivers for each forwarder. Conversely, a single receiver usually
receives data from many forwarders.
The receiver is either a Splunk Enterprise indexer (the typical case) or another
forwarder (referred to as an "intermediate forwarder") configured to receive data
from forwarders.
You must set up the receiver first. You can then set up forwarders to send data to
that receiver.
Set up receiving
Before enabling a Splunk Enterprise instance (either an indexer or a forwarder)
as a receiver, you must install it. You can then enable receiving on the instance
through Splunk Web, the CLI, or the inputs.conf configuration file.
Set up receiving with Splunk Web
Use Splunk Web to set up a receiver:
1. Log into Splunk Web as admin on the server that will be receiving data from a
forwarder.
2. Click the Settings link at the top of the page.
18
For <port>, substitute the port you want the receiver to listen on (the receiving
port). For example, if you enter "9997," the receiver will receive data on port
9997. By convention, receivers listen on port 9997, but you can specify any
unused port. You can use a tool like netstat to determine what ports are
available on your system. Make sure the port you select is not in use by
splunkweb or splunkd.
The splunk enable listen command creates a [splunktcp] stanza in
inputs.conf. For example, if you set the port to "9997", it creates the stanza
[splunktcp://9997].
Set up receiving with the configuration file
You can enable receiving on your Splunk Enterprise instance by configuring
inputs.conf in $SPLUNK_HOME/etc/system/local. To configure a universal
forwarder as an intermediate forwarder (a forwarder that functions also as a
receiver), use this method.
To enable receiving, add a [splunktcp] stanza that specifies the receiving port.
In this example, the receiving port is 9997:
[splunktcp://9997]
disabled = 0
19
In summary, you only need to install the app for the forwarder's OS on the
receiver (or search head) if it will be performing searches on the forwarded OS
20
data.
For <port>, substitute the port you want the receiver to listen on. This also
known as the "receiver port".
22
For <host>:<port>, substitute the host and receiver port number of the receiver.
For example, splunk_indexer.acme.com:9995.
Alternatively, if you have many forwarders, you can use an outputs.conf file to
specify the receiver. For example:
[tcpout:my_indexers]
server= splunk_indexer.acme.com:9995
You can create this file once, then distribute copies of it to each forwarder.
Steps to deployment
Once you have downloaded the universal forwarder and planned your
deployment, perform these steps:
1. Install the universal forwarder (with optional migration and configuration).
2. Test the deployment.
3. Perform additional configuration.
25
If your monitoring needs require you to install the universal forwarder to collect
remote Windows data, then you must configure your Windows environment for
the proper installation of the forwarder.
1. Create and configure security groups with the user you want the universal
forwarder to run as.
2. Optionally, configure the universal forwarder account as a managed service
account.
3. Create and configure Group Policy objects for security policy and user rights
assignment.
4. Assign appropriate user rights to the GPO.
5. Deploy the GPO(s) with the updated settings to the appropriate objects.
Note: These steps are high-level procedures only. For step-by-step instructions,
read "Prepare your Windows network for a Splunk Enterprise installation as a
network or domain user" in the Installation Manual. Depending on whether or not
you install the forwarder in low-privilege mode, one or more steps might not be
necessary.
If you attempt to run the installer in such a way, it warns you and prevents the
installation.
A series of dialogs guides you through the installation. When you're through with
a dialog, click Next to move to the next in the series. Here are the dialogs, in
order:
1. "Universal forwarder setup" dialog
To continue the installation, check the "Check this box to accept the License
Agreement" checkbox. To view the license agreement, click the "View License
Agreement" button.
Installation Options
New for version 6.2 of the universal forwarder, the Windows installer gives you
two choices: Install with the default installation settings, or configure all settings
prior to installing.
The installer does the following by default:
Installs the universal forwarder in \Program
Files\SplunkUniversalForwarder on the system drive (the drive that
booted your Windows system.)
Installs the universal forwarder with the default management port.
Configures the universal forwarder to run as the Local System user. Read
"Choose the user Splunk Enterprise should run as" in this manual to
understand the ramifications.
Enables the Application, System, and Security Windows Event Log inputs.
2a. If you want to change any of these default installation settings, click the
"Customize Options" button and proceed with the instructions in "Customize
Options" in this topic.
28
2b. Otherwise, click the "Install" button to install the software with the defaults.
Then, continue with Step 8.
Customize Options
On each panel, click Next to continue, Back to go back a step, or Cancel to
cancel the installation and quit the installer.
3. "Destination Folder" dialog
Select an SSL certificate for verifying the identity of this machine. This step is
optional. Skip this step if using Splunk Cloud.
Depending on your certificate requirements, you might need to specify a
password and a Root Certificate Authority (CA) certificate to verify the identity of
the certificate. If not, these fields can be left blank.
29
This step in the installer requires one or two dialogs, depending on the user type
you choose.
In the first dialog, specify whether you want the universal forwarder to run as the
Local System user or a domain user. The installer uses this information to
determine the permissions the universal forwarder needs.
If you select Local System, the universal forwarder installs as the Local System
user. This is recommended for improved security, unless you want this universal
forwarder to collect event logs or metrics from remote machines.
For more help in determining what to select here, see "Before you install" earlier
in this topic.
After you make your choice, click Next.
If you specify Local System, the installer skips the second screen and takes you
directly to the "Enable Windows Inputs" dialog.
If you specify Domain account, the installer takes you to a second dialog, where
you need to enter domain and user information for this instance of the universal
forwarder. The universal forwarder will run as the user you specify in this dialog.
Important: You must specify the user name in domain\username format. Failure
to include the domain name when specifying the user will cause the installation to
30
fail.
On the second dialog, at the bottom, there is a checkbox labeled "Add user as
local administrator". When the checkbox is checked (the default), the installer
adds the domain user you specified to the local Administrators group. When the
checkbox is not checked, the universal forwarder installs in "low-privilege" mode.
This mode is available for customers that cannot or do not want to run programs
as an administrator on servers. Read "Run the universal forwarder in
low-privilege mode" later in this topic for additional information and caveats.
To enable a normal installation as a user with local administrative privileges,
leave the box checked.
Important: In many cases, the user you specify must have specific rights
assigned to it prior to completing the installation. Failure to do so might result in a
failed installation. Read "Before you install" earlier in this topic for specific
information and links to step-by-step instructions.
Note: This dialog only appears if you previously specified a receiving indexer (in
the previous step).
6a. "Enable Windows Inputs" dialog
31
If you select any of the Windows inputs that the installer dialog shows you, the
installer brings up the "Choose the Splunk Add-on for Windows" dialog.
In this dialog:
Choose "Install the SPlunk Add-on for Microsoft Windows included with
this installer" if you do not already have a copy of the Splunk Add-on for
Windows installed on the local machine. Or,
Choose "Install an existing local copy of the Splunk Add-on for Microsoft
Windows" if you have a local copy of the add-on installed on the machine,
or if you have downloaded a more recent version from Splunkbase].
If you chose "Install an existing copy" above, locate the installed copy on your
system by clicking the Browse button.
Once you have completed your selection, click Next.
Note: This dialog only appears if you previously selected an input in the input
selection page.
7. "Specify a Deployment Server" dialog
Enter the hostname or IP address and management port for your deployment
server. The default management port is 8089. Skip this step if using Splunk
Cloud, unless you have an on-premises Deployment Server.
You can use the deployment server to push configuration updates to the
32
Enter the hostname or IP address and receiving port of the receiving indexer
(receiver). For information on setting up a receiver, see "Enable a receiver". Skip
this step if using Splunk Cloud.
Note: This step is optional, but if you skip it, you should enter a deployment
server in step 5; otherwise, the universal forwarder does not do anything, as it
does not have any way of determining which indexer to forward to. A popup
message appears which notes this. You can configure the forwarder with
configuration files later.
9. "Ready to Install the Program" dialog
33
35
2. Next, use the Add or Remove Programs control panel to uninstall the
forwarder. On Windows 7, 8, Server 2008, and Server 2012, that option is
available under Programs and Features.
Note: Under some circumstances, the Microsoft installer might present a reboot
prompt during the uninstall process. You can safely ignore this request without
rebooting.
36
Steps to deployment
Once you have downloaded the universal forwarder and have planned your
deployment, perform these steps:
1. Install the universal forwarder (with optional configuration).
2. Test and tune the deployment.
3. Perform any post-installation configuration.
4. Deploy the universal forwarder across your environment.
You can also install the forwarder as a user who is not an administrator on the
local machine. Use the SET_ADMIN_USER installation flag to install the forwarder in
"low privilege" mode.
If you install the forwarder as the Local System user, the forwarder can collect
any kind of data that is available on the local machine. It cannot, however, collect
data from other machines. This is by design.
You must give the universal forwarder a user account if you intend to do any of
the following:
Read Event Logs remotely
Collect performance counters remotely
Read network shares for log files
Enumerate the Active Directory schema, using Active Directory monitoring
Read "Choose the Windows user Splunk should run as" in the Installation
Manual for concepts and procedures on the user requirements that must be in
place before you collect remote Windows data.
Important: You must choose - and configure - the user that Splunk runs as
before attempting to install a universal forwarder for remote Windows data
collection. Failure to do so can result in a failed installation.
Configure your Windows environment prior to installation
To configure your Windows environment for the proper installation of the
forwarder, follow these steps:
1. Create and configure security groups with the user you want the universal
forwarder to run as.
2. Optionally, configure the universal forwarder account as a managed service
account.
3. Create and configure Group Policy or Local Security Policy objects for user
rights assignments.
4. Assign appropriate security settings.
5. If using Active Directory, deploy the Group Policy object(s) with the updated
settings to the appropriate objects.
38
Note: These steps are high-level procedures only. For step-by-step instructions,
read "Prepare your Windows network for a Splunk Enterprise installation as a
network or domain user" in the Installation Manual.
msiexec.exe /i splunkuniversalforwarder-<...>-x86-release.msi
[<flag>]... [/quiet]
msiexec.exe /i splunkuniversalforwarder-<...>-x64-release.msi
[<flag>]... [/quiet]
The value of <...> varies according to the particular release; for example,
splunkuniversalforwarder-4.2-86454-x64-release.msi.
Important: We do not recommend that you run the 32-bit version of the universal
forwarder on a 64-bit platform.
Command line flags allow you to configure your forwarder at installation time.
Using command line flags, you can specify a number of settings, including:
The user the universal forwarder runs as. (Be sure the user you specify
has the appropriate permissions to access the content you want to
forward.)
Whether or not the forwarder runs in "low-privilege" mode - as a user who
does not have local administrative access.
The receiving Splunk Enterprise instance that the universal forwarder will
send data to.
A deployment server for updating the configuration.
The Windows event logs to index.
Whether the universal forwarder should start automatically when the
installation is completed.
The following sections list the flags available and provide a few examples of
various configurations.
39
Default
AGREETOLICENSE=Yes|No
Specifies the
installation
directory.
Important: Do not
c:\Program
install the
Files\SplunkUniversalF
universal
forwarder over an
existing installation
of full Splunk
Enterprise.
INSTALLDIR="<directory_path>"
LOGON_USERNAME="<domain\username>"
LOGON_PASSWORD="<pass>"
SplunkForwarder
RECEIVING_INDEXER="<host:port>"
For information on
setting up a
receiver, see
"Enable a
receiver". Note:
This flag is
optional, but if you
don't specify it and
also don't specify
DEPLOYMENT_SERVER,
the universal
forwarder will be
unable to function,
41
n/a
RECEIVING_INDEXER,
the universal
forwarder will be
unable to function,
as it will not have
any way of
determining which
indexer to forward
to.
Use this flag to
specify whether
the universal
forwarder should
be configured to
launch
automatically
when the
installation
finishes.
LAUNCHSPLUNK=1|0
42
1 (yes)
SERVICESTARTTYPE=auto|manual
SERVICESTARTTYPE
auto
MONITOR_PATH="<directory_path>"
n/a
WINEVENTLOG_APP_ENABLE=1|0
WINEVENTLOG_SEC_ENABLE=1|0
application
WINEVENTLOG_SYS_ENABLE=1|0
security
WINEVENTLOG_FWD_ENABLE=1|0
system
WINEVENTLOG_SET_ENABLE=1|0
forwarders
setup
43
PERFMON=<input_type>,<input_type>,...
n/a
cpu memory
network diskspace
ENABLEADMON=1|0
CERTFILE=<c:\path\to\certfile.pem>
ROOTCACERTFILE=<c:\path\to\rootcacertfile.pem>
CERTPASSWORD=<password>
0 (not enabled)
set
RECEIVING_INDEXER
Deletes any
instance-specific
data in preparation
for creating a
0 (do not prepare the ins
clone of a
cloning.)
machine. This
invokes the splunk
CLONEPREP=1|0
clone-prep
SET_ADMIN_USER=1|0
45
Important: This
flag requires that
you set both the
LOGON_USERNAME
and
LOGON_PASSWORD
Examples
The following are some examples of using different flags.
Install the universal forwarder to run as the Local System user and request
configuration from deploymentserver1
You might do this for new deployments of the forwarder.
msiexec.exe /i splunkuniversalforwarder_x86.msi
DEPLOYMENT_SERVER="deploymentserver1:8089" AGREETOLICENSE=Yes /quiet
Install the universal forwarder to run as a domain user, but do not launch it
immediately
You might do this when preparing a sample host for cloning.
msiexec.exe /i splunkuniversalforwarder_x86.msi
LOGON_USERNAME="AD\splunk" LOGON_PASSWORD="splunk123"
DEPLOYMENT_SERVER="deploymentserver1:8089" LAUNCHSPLUNK=0
AGREETOLICENSE=Yes /quiet
msiexec.exe /i splunkuniversalforwarder_x86.msi
RECEIVING_INDEXER="indexer1:9997" WINEVENTLOG_SEC_ENABLE=1
WINEVENTLOG_SYS_ENABLE=1 AGREETOLICENSE=Yes /quiet
Note: You can also use the Services MMC snap-in (Start > Administrative
Tools > Services) to stop the SplunkForwarder service.
2. Next, run the Microsoft Installer to perform the uninstall:
msiexec /uninstall|x splunkuniversalforwarder-<...>-x86-release.msi
The installer has one supported flag that you can use during uninstallation:
Flag
REMOVE_FROM_GROUPS=1|0
49
Default
1 (Take away
elevated rights
and group
membership
on uninstall.)
Note: Under some circumstances, the Microsoft installer might present a reboot
prompt during the uninstall process. You can safely ignore this request without
rebooting.
Steps to deployment
Once you have downloaded the universal forwarder and have planned your
deployment, as described in "Universal forwarder deployment overview", perform
these steps:
1. Install and configure the universal forwarder on a test machine, using the
command line interface with the desired flags.
2. Test and tune the deployment.
3. Load the universal forwarder MSI into your deployment tool, specifying the
tested flags.
4. Execute deployment with your deployment tool.
5. Review log files on the forwarder to confirm that it has connected to the
receiving indexer.
50
Example installation
This example sets the universal forwarder to run as Local System user, get
inputs from Windows security and system event logs, send data to indexer1, and
launch automatically:
msiexec.exe /i splunkuniversalforwarder_x86.msi
RECEIVING_INDEXER="indexer1:9997" WINEVENTLOG_SEC_ENABLE=1
WINEVENTLOG_SYS_ENABLE=1 AGREETOLICENSE=Yes /quiet
51
Steps to deployment
Once you have downloaded the universal forwarder and have planned your
deployment, perform these steps:
1. Install the universal forwarder.
2. Configure (and optionally migrate) the universal forwarder.
3. Test the deployment.
4. Perform any additional configuration.
5. Deploy the universal forwarder across your environment.
Install on FreeBSD
Install on AIX
Install on HP-UX
Sun SPARC systems that run Solaris require a minimum patch level to
install a universal forwarder
If you plan to install a universal forwarder on a Sun SPARC system that runs
Solaris, confirm that you have patch level SUNW_1.22.7 or later of the C library
(libc.so.1). If you do not, the universal forwarder cannot run because it needs
this version of the library.
Installation procedure
You install the universal forwarder the same way that you install a full Splunk
Enterprise instance, as documented in these topics in the Installation manual.
There are only two differences:
The package name.
The default installation directory.
The package name
When you install a package, substitute the name of the universal forwarder
package for the full Splunk Enterprise package name used in the commands in
the Installation manual.
For example, if installing the universal forwarder onto Red Hat Linux, use this
command:
rpm -i splunkforwarder_<package_name>.rpm
rpm -i splunk_<package_name>.rpm
The only difference is the prefix to the package name: "splunkforwarder", instead
of "splunk".
The default install directory
The universal forwarder installs by default in the /opt/splunkforwarder directory.
(The default installation directory for full Splunk is /opt/splunk.)
53
splunk start
Accept the license agreement automatically
The first time you start the universal forwarder after a new installation, you must
accept the license agreement. To start the universal forwarder and accept the
license in one step:
where:
<host> is the deployment server's hostname or IP address and <port> is
the port it's listening on.
This step also automatically enables the deployment client functionality.
3. Configure the universal forwarder to forward to a specific receiving indexer,
also known as the "receiver" (optional):
where:
<host> is the receiving indexer's hostname or IP address and <port> is the
port it's listening on. By convention, the receiver listens for forwarders on
port 9997, but it can be set to listen on any port, so you'll need to check
with the receiver's administrator to obtain the port number. For information
on setting up a receiver, see "Enable a receiver".
<username>:<password> is the username and password for logging into the
forwarder. By default, these are "admin:changeme" (To set a different
password than the default , issue the following command "splunk edit user
admin -password <new password> -role admin -auth admin:changeme").
During this step, you can also configure a certificate for secure intra-Splunk
communications, using a set of optional ssl flags to specify a certificate, root CA,
and password. For example:
55
4. To configure the universal forwarder's inputs, use the CLI add command or edit
inputs.conf. See "About the CLI" and subsequent topics for details on using the
CLI.
For a complete list of CLI commands supported in the universal forwarder, see
"Supported CLI commands".
$SPLUNK_HOME/var/log/splunk/splunkd.log
$SPLUNK_HOME/var/log/splunk/metrics.log
$SPLUNK_HOME/var/log/splunk/license_audit.log
Steps to deployment
Once you have downloaded the universal forwarder and have planned your
deployment, as described in "Universal forwarder deployment overview", perform
these steps:
1. Install and configure the universal forwarder on a test machine, as described in
"Deploy a nix universal forwarder manually".
2. Test and tune the configuration.
3. Create a script wrapper for the installation and configuration commands.
57
4. Run the script on representative target machines to verify that it works with all
required shells.
5. Execute the script against the desired set of hosts.
6. Review log files on the forwarder to confirm that it has connected to the
receiving indexer.
#!/bin/sh
# This script provides an example of how to deploy the universal
forwarder
# to many remote hosts via ssh and common Unix commands.
#
# Note that this script will only work unattended if you have SSH host
keys
# setup & unlocked.
# To learn more about this subject, do a web search for "openssh key
management".
# ----------- Adjust the variables below ----------# Populate this file with a list of hosts that this script should
install to,
# with one host per line. You may use hostnames or IP addresses, as
# applicable. You can also specify a user to login as, for example,
"foo@host".
#
# Example file contents:
# server1
# server2.foo.lan
# you@server3
# 10.2.3.4
HOSTS_FILE="/path/to/splunk.install.list"
# This is the path to the tar file that you wish to push out. You may
# wish to make this a symlink to a versioned tar file, so as to minimize
# updates to this script in the future.
SPLUNK_FILE="/path/to/splunk-latest.tar.gz"
# This is where the tar file will be stored on the remote host during
# installation. The file will be removed after installation. You
normally will
# not need to set this variable, as $NEW_PARENT will be used by default.
#
59
# SCRATCH_DIR="/home/your_dir/temp"
# The location in which to unpack the new tar file on the destination
# host. This can be the same parent dir as for your existing
# installation (if any). This directory will be created at runtime, if
it does
# not exist.
NEW_PARENT="/opt"
# After installation, the forwarder will become a deployment client of
this
# host. Specify the host and management (not web) port of the
deployment server
# that will be managing these forwarder instances. If you do not wish
to use
# a deployment server, you may leave this unset.
#
# DEPLOY_SERV="splunkDeployMaster:8089"
# A directory on the current host in which the output of each
installation
# attempt will be logged. This directory need not exist, but the user
running
# the script must be able to create it. The output will be stored as
# $LOG_DIR/<[user@]destination host>. If installation on a host fails,
a
# corresponding file will also be created, as
# $LOG_DIR/<[user@]destination host>.failed.
LOG_DIR="/tmp/splunkua.install"
# For conversion from normal Splunk Enterprise installs to the universal
forwarder:
# After installation, records of progress in indexing files (monitor)
# and filesystem change events (fschange) can be imported from an
existing
# Splunk Enterprise (non-forwarder) installation. Specify the path to
that installation here.
# If there is no prior Splunk Enterprise instance, you may leave this
variable empty ("").
#
# NOTE: THIS SCRIPT WILL STOP THE SPLUNK ENTERPRISE INSTANCE SPECIFIED
HERE.
#
# OLD_SPLUNK="/opt/splunk"
# If you use a non-standard SSH port on the remote hosts, you must set
this.
# SSH_PORT=1234
# You must remove this line, or the script will refuse to run.
60
This is
to
# ensure that all of the above has been read and set. :)
UNCONFIGURED=1
# ----------- End of user adjustable settings -----------
# helpers.
faillog() {
echo "$1" >&2
}
fail() {
faillog "ERROR: $@"
exit 1
}
# error checks.
test "$UNCONFIGURED" -eq 1 && \
fail "This script has not been configured. Please see the notes in
the script."
test -z "$HOSTS_FILE" && \
fail "No hosts configured! Please populate HOSTS_FILE."
test -z "$NEW_PARENT" && \
fail "No installation destination provided! Please set NEW_PARENT."
test -z "$SPLUNK_FILE" && \
fail "No splunk package path provided! Please populate SPLUNK_FILE."
if [ ! -d "$LOG_DIR" ]; then
mkdir -p "$LOG_DIR" || fail "Cannot create log dir at \"$LOG_DIR\"!"
fi
# some setup.
if [ -z "$SCRATCH_DIR" ]; then
SCRATCH_DIR="$NEW_PARENT"
fi
if [ -n "$SSH_PORT" ]; then
SSH_PORT_ARG="-p${SSH_PORT}"
SCP_PORT_ARG="-P${SSH_PORT}"
fi
NEW_INSTANCE="$NEW_PARENT/splunkforwarder" # this would need to be
edited for non-UA...
DEST_FILE="${SCRATCH_DIR}/splunk.tar.gz"
#
#
# create script to run remotely.
#
61
#
REMOTE_SCRIPT="
fail() {
echo ERROR: \"\$@\" >&2
test -f \"$DEST_FILE\" && rm -f \"$DEST_FILE\"
exit 1
}
"
###
try untarring tar file.
REMOTE_SCRIPT="$REMOTE_SCRIPT
(cd \"$NEW_PARENT\" && tar -zxf \"$DEST_FILE\") || fail \"could not
untar /$DEST_FILE to $NEW_PARENT.\"
"
###
setup seed file to migrate input records from old instance, and
stop old instance.
if [ -n "$OLD_SPLUNK" ]; then
REMOTE_SCRIPT="$REMOTE_SCRIPT
echo \"$OLD_SPLUNK\" > \"$NEW_INSTANCE/old_splunk.seed\" || fail
\"could not create seed file.\"
\"$OLD_SPLUNK/bin/splunk\" stop || fail \"could not stop existing
splunk.\"
"
fi
###
setup deployment client if requested.
if [ -n "$DEPLOY_SERV" ]; then
REMOTE_SCRIPT="$REMOTE_SCRIPT
\"$NEW_INSTANCE/bin/splunk\" set deploy-poll \"$DEPLOY_SERV\"
--accept-license --answer-yes \
--auto-ports --no-prompt || fail \"could not setup deployment
client\"
"
fi
###
start new instance.
REMOTE_SCRIPT="$REMOTE_SCRIPT
\"$NEW_INSTANCE/bin/splunk\" start --accept-license --answer-yes
--auto-ports --no-prompt || \
fail \"could not start new splunk instance!\"
"
###
remove downloaded file.
REMOTE_SCRIPT="$REMOTE_SCRIPT
rm -f "$DEST_FILE" || fail \"could not delete downloaded file
$DEST_FILE!\"
"
#
#
62
63
64
You can migrate checkpoint data from an existing Windows light forwarder
(version 4.0 or later) to the universal forwarder. For an overview of migration, see
"Migrate from a light forwarder" in this manual.
If you want to migrate, you must do so during the installation process. You
cannot migrate post-installation.
You perform a Windows installation with either the installer GUI or the command
line:
If you use the installer GUI, one of the screens will prompt you to
migrate. See "Deploy a Windows universal forwarder via the installer GUI"
for a walkthrough of the GUI installation procedure.
If you install via the command line, the flag MIGRATESPLUNK=1 specifies
migration. See "Deploy a Windows universal forwarder via the command
line" for a list of supported flags and how to use them to configure your
installation.
Important: You must install the universal forwarder in a different directory from
the existing light forwarder. Since the default install directory for the universal
forwarder is C:\Program Files\SplunkUniversalForwarder and the default install
directory for full Splunk Enterprise (including the light forwarder) is C:\Program
Files\Splunk, you'll be safe if you just stick with the defaults.
What the installer does
Whichever installation method you use, the Windows installer performs the
following actions:
1. Searches for an existing heavy or light forwarder on the machine.
2. Determines whether the forwarder is eligible for migration (must be at version
4.0 or above).
3. If it finds an eligible forwarder, the GUI offers the user the option of migrating.
(The command line installer looks to see whether the MIGRATESPLUNK=1 flag
exists.)
4. If user specifies migration (or the MIGRATESPLUNK=1 flag exists), the installer
shuts down any running services (splunkd and, if running, splunkweb) for the
existing forwarder. It also sets the startup type of the services to manual, so that
they don't start up again upon reboot.
65
$SPLUNK_HOME/bin/splunk stop
Important: Make sure you install the universal forwarder into a different directory
from the existing light forwarder. Since the default install directory for the
universal forwarder is /opt/splunkforwarder and the default install directory for
full Splunk Enterprise (including the light forwarder) is /opt/splunk, you'll be safe
if you just stick with the defaults.
3. In the universal forwarder's installation directory, (the new $SPLUNK_HOME),
create a file named old_splunk.seed; in other words:
$SPLUNK_HOME/old_splunk.seed. This file must contain a single line, consisting of
the path of the old forwarder's $SPLUNK_HOME directory. For example:
/opt/splunk.
4. Start the universal forwarder:
$SPLUNK_HOME/bin/splunk start
The universal forwarder will migrate the checkpoint files from the forwarder
specified in the $SPLUNK_HOME/old_splunk.seed file. Migration only occurs the
first time you run the start command. You can leave the old_splunk.seed in
place; it only gets examined the first time you start the forwarder after installing it.
5. Perform any additional configuration of the universal forwarder, as described in
"Deploy a nix universal forwarder manually." Since the migration process only
copies checkpoint files, you will probably want to manually copy over the old
forwarder's inputs.conf configuration file (or at least examine it, to determine
what data inputs it was monitoring).
Once the universal forwarder is up and running (and after you've tested to ensure
migration worked correctly), you can uninstall the old forwarder.
67
68
On *nix systems: From a shell prompt on the host, run this command:
# splunk restart
69
70
Custom versions
When you configure forwarding behavior, those changes get saved in custom
versions of outputs.conf. There are several ways you can specify forwarding
behavior:
While installing the forwarder (Windows universal forwarder only)
By running CLI commands
By using Splunk Web (heavy/light forwarders only)
By directly editing an outputs.conf file
The forwarder automatically creates or edits custom versions of outputs.conf in
response to the first three methods. The locations of those versions vary,
depending on the type of forwarder and other factors:
The universal forwarder. If you use the CLI to make changes to
universal forwarder output behavior, it creates or edits a copy of
outputs.conf in $SPLUNK_HOME/etc/system/local. However, the Windows
installation process writes configuration changes to an outputs.conf file
located in the MSICreated app. For more information on configuring the
universal forwarder, see "Configure the universal forwarder."
Heavy and light forwarders. When you enable a heavy/light forwarder
through Splunk Web or the CLI, an outputs.conf file gets created in the
directory of the currently running app. For example, if you're working in the
search app, the file is created in $SPLUNK_HOME/etc/apps/search/local/.
You can then edit it there.
In addition to any outputs.conf files that you create and edit indirectly (for
example, through the CLI), you can also create or edit an outputs.conf file
directly. It is recommended that you work with just a single copy of the file, which
you place in $SPLUNK_HOME/etc/system/local/. (If a copy of the file already
exists in that directory, because of configuration changes made through the CLI,
just edit that copy.) For purposes of distribution and management simplicity, you
can combine settings from all non-default versions into a single custom
outputs.conf file.
After making changes to outputs.conf, you must restart the forwarder for the
changes to take effect.
For detailed information on outputs.conf, look here for the spec and examples.
71
Configuration levels
There are two types of output processors: tcpout and syslog. You can configure
them at three levels of stanzas:
Global. At the global level, you specify any attributes that you want to
apply globally, as well as certain attributes only configurable at the
system-wide level for the output processor. This stanza is optional.
Target group. A target group defines settings for one or more receiving
indexers. There can be multiple target groups per output processor. Most
configuration settings can be specified at the target group level.
Single server. You can specify configuration values for single servers
(receivers) within a target group. This stanza type is optional.
Configurations at the more specific level take precedence. For example, if you
specify compressed=true for a target group, the forwarder will send the servers in
that target group compressed data, even if compressed is set to "false" for the
global level.
Note: This discussion focuses on the tcpout processor, which uses the [tcpout]
header. For the syslog output processor, see "Forward data to third-party
systems" for details.
Global stanza
Here you set any attributes that you want to apply globally. This stanza is not
required. However, there are several attributes that you can set only at the global
level, including defaultGroup and indexAndForward.
The global stanza for the tcpout procesor is specified with the [tcpout] header.
Here's an example of a global tcpout stanza:
[tcpout]
defaultGroup=indexer1
indexAndForward=true
as well as forward the data to receiving indexers in the target groups. If set
to "false" (the default), the forwarder just forwards data but does not index
it. This attribute is only available for heavy forwarders; universal and light
forwarders cannot index data.
Default target groups
To set default groups for automatic forwarding, include the defaultGroup attribute
at the global level, in your [tcpout] stanza:
[tcpout]
defaultGroup= <target_group1>, <target_group2>, ...
[tcpout:<target_group>]
server=<receiving_server1>, <receiving_server2>, ...
<attribute1> = <val1>
<attribute2> = <val2>
...
73
See "Define typical deployment topologies", later in this topic, for information on
how to use the target group stanza to define several deployment topologies.
Single-server stanza
You can define a specific configuration for an individual receiving indexer.
However, the receiver must also be a member of a target group.
When you define an attribute at the single-server level, it takes precedence over
any definition at the target group or global level.
Here is the syntax for defining a single-server stanza:
[tcpout-server://<ipaddress_or_servername>:<port>]
<attribute1> = <val1>
<attribute2> = <val2>
...
Example
The following outputs.conf example contains three stanzas for sending tcpout to
Splunk Enterprise receivers:
Global settings. In this example, there is one setting, to specify a
defaultGroup.
Settings for a single target group consisting of two receivers. Here, we are
specifying a load-balanced target group consisting of two receivers.
Settings for one receiver within the target group. In this stanza, you can
specify any settings specific to the mysplunk_indexer1 receiver.
[tcpout]
defaultGroup=my_indexers
[tcpout:my_indexers]
server=mysplunk_indexer1:9997, mysplunk_indexer2:9996
[tcpout-server://mysplunk_indexer1:9997]
Load balancing
To perform load balancing, specify one target group with multiple receivers. In
this example, the target group consists of three receivers:
[tcpout:my_LB_indexers]
server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995
The forwarder will load balance between the three receivers listed. If one receiver
goes down, the forwarder automatically switches to the next one available.
Data cloning
To perform data cloning, specify multiple target groups, each in its own stanza.
In data cloning, the forwarder sends copies of all its events to the receivers in two
or more target groups. Data cloning usually results in similar, but not necessarily
exact, copies of data on the receiving indexers. Here's an example of how you
set up data cloning:
[tcpout]
defaultGroup=indexer1,indexer2
[tcpout:indexer1]
server=10.1.1.197:9997
[tcpout:indexer2]
server=10.1.1.200:9997
The forwarder will send duplicate data streams to the servers specified in both
the indexer1 and indexer2 target groups.
Data cloning with load balancing
You can combine load balancing with data cloning. For example:
[tcpout]
defaultGroup=cloned_group1,cloned_group2
[tcpout:cloned_group1]
server=10.10.10.1:9997, 10.10.10.2:9997, 10.10.10.3:9997
[tcpout:cloned_group2]
server=10.1.1.197:9997, 10.1.1.198:9997, 10.1.1.199:9997,
10.1.1.200:9997
75
The forwarder will send full data streams to both cloned_group1 and
cloned_group2. The data will be load-balanced within each group, rotating
among receivers every 30 seconds (the default frequency).
Note: For syslog and other output types, you must explicitly specify routing as
described here: "Route and filter data".
defaultGroup
indexAndForward
server
Default
n/a
false
n/a
Where
configured
global
stanza
global
stanza
Value
A comma-separated list of one or
more target groups. Forwarder
sends all events to all specified
target groups. Don't set this
attribute if you don't want events
automatically forwarded to a target
group.
If set to "true", the forwarder will
index all data locally, in addition to
forwarding the data to a receiving
indexer.
Important: This attribute is only
available for heavy forwarders. A
universal forwarder cannot index
locally.
<ipaddress_or_servername>:<port>,
where <port> is the receiving
false
any stanza
level
76
there.
sendCookedData
true
global or
Specifies whether data is cooked
target group
before forwarding.
stanza
compressed
false
global or
Specifies whether the forwarder
target group
sends compressed data.
stanza
ssl....
useACK
n/a
any stanza
level
false
$SPLUNK_HOME/etc/apps/<appname>/local/inputs.conf.
Edit inputs.conf
Editing inputs.conf on a universal forwarder is identical to editing inputs.conf on a
full Splunk instance:
1. Using your operating system file management tools or a shell or command
prompt, avigate to $SPLUNK_HOME/etc/system/local.
2. Open inputs.conf for editing. You might need to create this file if it does not
exist.
3. Add your data inputs by specifying input stanzas. See "What Splunk
Enterprise can index" and "Edit inputs.conf."
4. Once you have defined your inputs, save the file and close it.
5. Restart the forwarder.
6. On the receiving indexer, log in and load the Search and Reporting app.
7. Run a search and confirm that you see results from the forwarder that you set
up the data inputs on:
host=<forwarder host name or ip address> source=<data source>
earliest=1h
If you don't see any results, visit the Troubleshooting page for possible
resolution.
79
add
app
config
datastore-dir
default-hostname
deploy-client
deploy-poll
eventlog
exec
forward-server
monitor
oneshot
perfmon
registry
servername
splunkd-port
tcp
udp
user
wmi
Note: A few commands, such as start and stop can be run without an object. A
command with no object is also valid for the universal forwarder.
80
As described above, it's the object that determines whether a command is valid
in the universal forwarder. For example, the above list includes the monitor
object. Therefore, the add monitor and edit monitor command/object
combinations are both valid. For more information on the monitor object, see
"Use the CLI to monitor files and directories" in the Getting Data In manual.
For more details on using the CLI in general, see the "Administer Splunk
Enterprise with the CLI" chapter in the Admin manual. In particular, the topic "CLI
admin commands" provides details on CLI syntax, including a list of all
commands supported by full Splunk Enterprise and the objects they can act
upon.
81
6. On the receivers, search for data to confirm that forwarding, along with any
configured behaviors like load balancing or routing, occurs as expected.
Enable a receiver
To enable forwarding and receiving, you configure both a receiver and a
forwarder. The receiver is the Splunk Enterprise instance receiving the data; the
forwarder sends data to the receiver.
Depending on your needs (for example to enable load balancing), you might
have multiple receivers for each forwarder. Conversely, a single receiver usually
receives data from many forwarders.
The receiver is either a Splunk Enterprise indexer (the typical case) or another
forwarder (referred to as an "intermediate forwarder") configured to receive data
from forwarders.
You must set up the receiver first. You can then set up forwarders to send data to
that receiver.
Set up receiving
Before enabling a Splunk Enterprise instance (either an indexer or a forwarder)
as a receiver, you must install it. You can then enable receiving on the instance
through Splunk Web, the CLI, or the inputs.conf configuration file.
Set up receiving with Splunk Web
Use Splunk Web to set up a receiver:
1. Log into Splunk Web as admin on the server that will be receiving data from a
forwarder.
2. Click the Settings link at the top of the page.
3. Select Forwarding and receiving in the Data area.
4. Click Add new in the Receive data section.
5. Specify which TCP port you want the receiver to listen on (the listening port,
also known as the receiving port). For example, if you enter "9997," the receiver
83
will receive data on port 9997. By convention, receivers listen on port 9997, but
you can specify any unused port. You can use a tool like netstat to determine
what ports are available on your system. Make sure the port you select is not in
use by splunkweb or splunkd.
6. Click Save. You must restart the instance to complete the process.
Set up receiving with Splunk CLI
To enable receiving, run the CLI command:
For <port>, substitute the port you want the receiver to listen on (the receiving
port). For example, if you enter "9997," the receiver will receive data on port
9997. By convention, receivers listen on port 9997, but you can specify any
unused port. You can use a tool like netstat to determine what ports are
available on your system. Make sure the port you select is not in use by
splunkweb or splunkd.
The splunk enable listen command creates a [splunktcp] stanza in
inputs.conf. For example, if you set the port to "9997", it creates the stanza
[splunktcp://9997].
Set up receiving with the configuration file
You can enable receiving on your Splunk Enterprise instance by configuring
inputs.conf in $SPLUNK_HOME/etc/system/local. To configure a universal
forwarder as an intermediate forwarder (a forwarder that functions also as a
receiver), use this method.
To enable receiving, add a [splunktcp] stanza that specifies the receiving port.
In this example, the receiving port is 9997:
[splunktcp://9997]
disabled = 0
84
In summary, you only need to install the app for the forwarder's OS on the
receiver (or search head) if it will be performing searches on the forwarded OS
data.
85
Set up forwarding
You can use Splunk Web or the CLI as a quick way to enable forwarding in a
Splunk Enterprise instance.
You can also enable, as well as configure, forwarding by creating an
outputs.conf file on the Splunk Enterprise instance. Although setting up
forwarders with outputs.conf requires a bit more initial knowledge, there are
obvious advantages to performing all forwarder configurations in a single
location. Most advanced configuration options are available only through
outputs.conf. In addition, if you will be enabling and configuring a number of
forwarders, you can easily accomplish this by editing a single outputs.conf file
86
and making a copy for each forwarder. See the topic "Configure forwarders with
outputs.conf" for more information.
Set up heavy forwarding with Splunk Web
You can use Splunk Web to set up a heavy forwarder:
1. Log into Splunk Web as admin on the server that will be forwarding data.
2. Click the Settings link at the top of the page.
3. Select Forwarding and receiving in the Data area.
4. Click Add new in the Forward data section.
5. Enter the hostname or IP address for the receiving Splunk Enterprise
instance(s), along with the receiving port specified when the receiver was
configured. For example, you might enter: receivingserver.com:9997. To
implement load-balanced forwarding, you can enter multiple hosts as a
comma-separated list.
6. Click Save. You must restart the instance to complete the process.
You can use Splunk Web to perform one other configuration. To store a copy of
indexed data local to the forwarder:
1. From Forwarding and receiving, select Forwarding defaults.
2. Select Yes to store and maintain a local copy of the indexed data on the
forwarder.
Important: A heavy forwarder has a key advantage over light and universal
forwarders in that it can index your data locally, as well as forward the data to
another index. However, local indexing is turned off by default. If you want to
store data on the forwarder, you must enable that capability - either in the
manner described above or by editing outputs.conf.
All other configuration must be done in outputs.conf.
Set up heavy forwarding with the CLI
With the CLI, setting up forwarding is a two step process. First you enable
forwarding on the Splunk Enterprise instance. Then you start forwarding to a
87
specified receiver.
To access the CLI, first navigate to $SPLUNK_HOME/bin/.
To enable the forwarder mode, enter:
Note: Although this command ends forwarding activity, the instance remains
configured as a forwarder. To revert the forwarder to a full Splunk Enterprise
instance, use the disable command, as described earlier in this topic.
Important: After invoking either of these commands, restart the forwarder.
Upgrade a forwarder
To upgrade a forwarder to a new version, just upgrade the instance in the usual
fashion. For details, read the upgrade section of the Installation manual.
88
Important: Before doing an upgrade, consider whether you really need to. In
many cases, there's no compelling reason to upgrade a forwarder. Forwarders
are always compatible with later version indexers, so you do not need to upgrade
them just because you've upgraded the indexers they're sending data to.
Back up your files first
Before you perform the upgrade, we strongly recommend that you back up all of
your files. Most importantly, back up your Splunk Enterprise configuration files.
For information on backing up configurations, read "Back up configuration
information" in the Admin manual.
If you're upgrading a heavy forwarder that's indexing data locally, you also need
to back up the indexed data. For information on backing up data, read "Back up
indexed data" in the Managing Indexers and Clusters of Indexers manual.
You cannot downgrade to a previous version; if you need to revert to an older
forwarder release, reinstall the instance.
Set up forwarding
You can use the CLI as a quick way to enable forwarding.
You can also enable, as well as configure, forwarding by creating an
outputs.conf file on the Splunk Enterprise instance. Although setting up
forwarders with outputs.conf requires a bit more initial knowledge, there are
obvious advantages to performing all forwarder configurations in a single
location. Most advanced configuration options are available only through
outputs.conf. In addition, if you will be enabling and configuring a number of
forwarders, you can easily accomplish this by editing a single outputs.conf file
and making a copy for each forwarder. See the topic "Configure forwarders with
outputs.conf" for more information.
Set up light forwarding with the CLI
With the CLI, setting up forwarding is a two step process. First you enable
forwarding on the instance. Then you start forwarding to a specified receiver.
To access the CLI, first navigate to $SPLUNK_HOME/bin/.
To enable the light forwarder mode, enter:
90
Note: Although this command ends forwarding activity, the instance remains
configured as a forwarder. To revert the instance to a full Splunk Enterprise
instance, use the disable command, as described earlier in this topic.
Important: After invoking either of these commands, restart the forwarder.
Upgrade a forwarder
To upgrade a forwarder to a new version, just upgrade the instance in the usual
fashion. For details, read the upgrade section of the Installation manual.
Important: Before doing an upgrade, consider whether you really need to. In
many cases, there's no compelling reason to upgrade a forwarder. Forwarders
are always compatible with later version indexers, so you do not need to upgrade
them just because you've upgraded the indexers they're sending data to.
Back up your files first
Before you perform the upgrade, we strongly recommend that you back up all of
your files. Most importantly, back up your configuration files. For information on
backing up configurations, read "Back up configuration information" in the Admin
91
manual.
[pipeline:distributedSearch]
disabled = true
For a detailed view of the exact configuration, see the configuration files for the
SplunkForwarder application in
$SPLUNK_HOME/etc/apps/SplunkForwarder/default.
[pipeline:indexerPipe]
disabled_processors= indexandforward, diskusage,
signing,tcp-output-generic-processor, syslog-output-generic-processor,
http-output-generic-processor, stream-output-processor
[pipeline:distributedDeployment]
disabled = true
[pipeline:distributedSearch]
disabled = true
[pipeline:fifo]
disabled = true
[pipeline:merging]
disabled = true
[pipeline:typing]
disabled = true
[pipeline:udp]
disabled = true
[pipeline:tcp]
disabled = true
[pipeline:syslogfifo]
disabled = true
[pipeline:syslogudp]
disabled = true
[pipeline:parsing]
disabled_processors=utf8, linebreaker, header, sendOut
[pipeline:scheduler]
disabled_processors = LiveSplunks
These modules include the deployment server (not the deployment client),
distributed search, named pipes/FIFOs, direct input from network ports, and the
scheduler.
93
The defaults for the light forwarder can be tuned to meet your needs by
overriding the settings in
$SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/default-mode.conf
on
a case-by-case basis.
Purge old indexes
When you convert an indexer instance to a light forwarder, among other things,
you disable indexing. In addition, you no longer have access to any data
previously indexed on that instance. However, the data still exists.
If you want to purge that data from your system, you must first disable the
SplunkLightForwarder app, then run the CLI clean command, and then renable
the app. For information on the clean command, see "Remove indexed data from
Splunk" in the Managing Indexers and Clusters of Indexers manual.
94
Upgrade forwarders
Upgrade the Windows universal forwarder
This topic describes the procedure for upgrading your Windows universal
forwarder from version 5.0.x, 6.0.x, 6.1.x, or 6.2.x to 6.3.
When you upgrade a universal forwarder, the installer performs an upgrade with
no configuration changes. If you need to change any configuration settings on
your forwarders, you can do so after the upgrade. A deployment server can
assist in the configuration update process.
This topic describes three upgrade scenarios:
Upgrade a single forwarder with the GUI installer
Upgrade a single forwarder with the command line installer
Perform a remote upgrade of a group of forwarders
For deployments of any size, you will most likely want to use this last scenario.
in %SPLUNK_HOME%\var\lib\modinputs\
2. Uninstall the existing 32-bit forwarder.
3. Install the 64-bit forwarder.
4. Restore your apps, configurations and checkpoints by copying them to the
appropriate directories:
%SPLUNK_HOME%\etc\system\local for configuration files.
%SPLUNK_HOME%\etc\apps for apps and add-ons.
%SPLUNK_HOME%\var\lib\modinputs for checkpoint files.
96
msiexec.exe /i splunkuniversalforwarder-<...>-x86-release.msi
[AGREETOLICENSE=Yes /quiet]
msiexec.exe /i splunkuniversalforwarder-<...>-x64-release.msi
[AGREETOLICENSE=Yes /quiet]
The value of <...> varies according to the particular release; for example,
splunkuniversalforwarder-5.0-142438-x64-release.msi.
Note: You cannot make configuration changes during an upgrade. The installer
ignores any command line flags that you specify except for
"AGREETOLICENSE".
3. The forwarder starts automatically when you complete the installation.
The installer puts a log of upgrade changes in the %TEMP% directory. It also reports
any errors in the Application Event Log.
97
1. Load the universal forwarder MSI into your deployment tool. Specify the
command like as follows:
See the previous section, "Upgrade using the command line", for details on the
MSI command.
2. Execute deployment with your deployment tool.
3. Verify that the universal forwarders function properly.
You might want to test the upgrade locally on one machine before performing a
remote upgrade across all your forwarders.
98
$SPLUNK_HOME/bin/splunk stop
Important: Make sure no other processes can start the forwarder automatically
(such as Solaris SMF).
2. Install the universal forwarder package over the existing deployment:
If you use a .tar file, expand it into the same directory with the same
ownership as the existing universal forwarder instance. This overwrites
and replaces matching files but does not remove unique files.
If you use a package manager, such as an RPM, type in rpm -U
<splunk_package_name>.rpm from a shell prompt.
If you use a .dmg file (on MacOS), double-click it and follow the
instructions. Be sure to specify the same installation directory as your
existing installation.
If you use init scripts, be sure to include the following so the End-User
License Agreement (EULA) gets accepted:
99
$SPLUNK_HOME/bin/splunk start
4. Choose whether you want to run the migration preview script to see what
changes will be made to your existing configuration files, or proceed with the
migration and upgrade right away.
5. If you choose to view the expected changes, the script provides a list.
6. Once you've reviewed these changes and are ready to proceed with migration
and upgrade, run $SPLUNK_HOME/bin/splunk start again.
Note: You can complete Steps 3 to 5 in one line:
To accept the license and view the expected changes (answer 'n') before
continuing the upgrade:
To accept the license and begin the upgrade without viewing the changes
(answer 'y'):
100
101
indexer open and continues to send the data stream until it has been safely sent.
Important: Universal forwarders are not able to switch indexers when monitoring
TCP network streams of data (including Syslog) unless an EOF is reached or an
indexer goes down, at which point the forwarder will switch to the next indexer in
the list. Because the universal forwarder does not parse the data and identify
event boundaries before forwarding the data to the indexer (unlike a heavy
forwarder), it has no way of knowing when it's safe to switch to the next indexer
unless it receives an EOF.
Note: Round-robin load balancing, which was previously available as an
alternative to automatic load balancing, was deprecated in Splunk Enterprise
version 4.2.
This diagram shows a typical load-balancing scenario, in which three forwarders
are sending load-balanced data across a set of two receiving indexers:
The main advantage of a static list is that it allows you to specify a different port
for each receiver. This is useful if you need to perform load balancing across
multiple receivers running on a single host. Each receiver can listen on a
separate port.
Static list target
To use a static list for the target, you simply specify each of the receivers in the
target group's [tcpout] stanza in the forwarder's outputs.conf file. In this
example, the target group consists of three receivers, specified by IP address
and receiver port number:
[tcpout: my_LB_indexers]
server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995
The universal forwarder will load balance between the three receivers listed. If
one receiver goes down, the forwarder automatically switches to another one on
the list.
DNS list target
To use a DNS list, edit your forwarder's outputs.conf file to specify a single host
in the target group's [tcpout] stanza. For example:
[tcpout:my_LB_indexers]
server=splunkreceiver.mycompany.com:9997
In your DNS server, create a DNS A record for each host's IP address,
referencing the server name you specified in outputs.conf. For example:
splunkreceiver.mycompany.com
splunkreceiver.mycompany.com
splunkreceiver.mycompany.com
A
A
A
10.10.10.1
10.10.10.2
10.10.10.3
The forwarder will use the DNS list to load balance, sending data in intervals,
switching among the receivers specified. If a receiver is not available, the
forwarder skips it and sends data to another one on the list.
If you have a topology with many forwarders, the DNS list method allows you to
update the set of receivers by making changes in just a single location, without
touching the forwarders' outputs.conf files.
104
splunkreceiver.mycompany.com
splunkreceiver.mycompany.com
splunkreceiver.mycompany.com
A
A
A
10.10.10.1
10.10.10.2
10.10.10.3
4. Create a single outputs.conf file for use by all the forwarders. This one
specifies the DNS server name used in the DNS list and the port the receivers
are listening on:
[tcpout]
defaultGroup=my_LB_indexers
[tcpout:my_LB_indexers]
disabled=false
autoLBFrequency=40
server=splunkreceiver.mycompany.com:9997
The steps are similar if you're using a static list instead of DNS.
./splunk
./splunk
./splunk
./splunk
add
add
add
add
forward-server
forward-server
forward-server
forward-server
indexer1:9997
indexer2:9997
indexer3:9997
indexer4:9997
-method
-method
-method
-method
autobalance
autobalance
autobalance
autobalance
Task Force (IETF) Request for Comments (RFC) Memo #1928. For information
on this memo, see "Network Working Group: Request for Comments: 1928"
(http://www.ietf.org/rfc/rfc1928.txt) on the IETF website.
Description
Default
socksUsername
socksPassword
108
[tcpout]
defaultGroup = proxy_indexers
[tcpout:proxy_indexers]
server = indexer1.slapstick.com:9997, indexer2.slapstick.com:9997
socksServer = prx.slapstick.com:1080
This example uses credentials to authenticate into the proxy host before
attempting to send data, and tells the proxy host to resolve DNS to determine the
indexers to connect for sending data:
[tcpout]
defaultGroup = socksCredentials
[tcpout:socksCredentials]
server = indexer3.slapstick.com:9997
socksServer = prx.slapstick.com:1081
socksUsername = proxysrv
socksPassword = letmein
socksResolveDNS = true
Security considerations
Note the following caveats when using this feature:
SOCKS5 proxy support only exists between the forwarder and the indexer
inclusive. There is no support for the usage of SOCKS with any other
Splunk Enterprise features, apps, or add-ons.
The SOCKS5 protocol sends authentication credentials in clear text. Due
to this implementation, these credentials are vulnerable to a
man-in-the-middle attacker. This means that an attacker can secretly relay
and possibly change communication between the SOCKS client and the
SOCKS proxy host. This is a caveat of the SOCKS protocol, not the
implementation of this feature in Splunk Enterprise.
For the most secure results, use the SOCKS attributes only on forwarders
which are inside networks that a SOCKS proxy host protects. Deploying a
forwarder in an unprotected environment can result in the interception of
109
110
5. Restart the forwarder, as described in "Start and stop Splunk Enterprise" in the
Admin manual.
You can repeat these steps to add more forwarders to the tier.
Configure forwarders to use the intermediate forwarding tier
To set up additional forwarders to send their data to the intermediate forwarding
tier:
1. If you have not already, install the universal forwarder.
2. Configure the forwarder to send data to the intermediate forwarder.
3. Configure local data inputs on the forwarder.
4. Restart the forwarder.
Test the configuration
To confirm that the intermediate tier works properly:
1. On the receiving indexer, sign into Splunk Enterprise.
2. Open the Search and Reporting app.
3. Run a search that contains a reference to one of the hosts that you configured
to send data to the intermediate forwarder:
host=<name or ip address of forwarder> index=_internal
If you do not see events, then the host has not been configured properly. See
"Troubleshoot forwarder/receiver connection" in this manual for possible fixes.
1. If you have not already, install the full Splunk Enterprise instance, as described
in "Installation instructions" in the Installation manual.
2. Use Splunk Web to configure the forwarder to send data to the receiving
indexer, as described in "Enable forwarding on a Splunk Enterprise instance."
3. Use Splunk Web to enable receiving on the instance, as described in "Enable
a receiver."
4. (Optional) Configure local data inputs on the forwarder. You can use Splunk
Web or edit configuration files.
5. (Optional) If you want to reduce the resource footprint of the forwarder,
configure the instance as a light forwarder.
Note: The light forwarder has been deprecated, and support for this feature
could be removed in a future release.
6. Restart the instance.
Configure forwarders to use the intermediate forwarding tier
To set up additional forwarders to send their data to the intermediate forwarding
tier:
1. If you have not already, install the universal or heavy forwarder.
2. Configure the forwarder to send data to the intermediate forwarder.
3. Configure local data inputs on the forwarder.
4. Restart the forwarder.
Test the configuration
To confirm that the intermediate tier works properly:
1. On the receiving indexer, sign into Splunk Enterprise.
2. Open the Search and Reporting app.
3. Run a search that contains a reference to one of the hosts that you configured
to send data to the intermediate forwarder:
112
If you do not see events, then the host has not been configured properly. See
"Troubleshoot forwarder/receiver connection" in this manual for possible fixes.
The acknowledgment tells the forwarder that the indexer received the data and
successfully wrote it to the file system. Upon receiving the acknowledgment, the
forwarder releases the block from memory.
If the wait queue is of sufficient size, it doesn't fill up while waiting for
acknowledgments to arrive. But see this section for possible issues and ways to
address them, including how to increase the wait queue size.
114
115
[tcpout:<target_group>]
server=<server1>, <server2>, ...
useACK=true
You can set maxQueueSize to specific values if necessary. See the outputs.conf
spec file for further details on maxQueueSize.
Note the following points regarding the maxQueueSize=auto recommendation:
When you turn on indexer acknowledgment, the increase in queue size
takes effect only after you restart the forwarder.
The auto setting is only available for forwarders of version 5.0.4 and
above. For earlier version forwarders running indexer acknowledgment,
you need to explicitly set the maxQueueSize attribute to 7MB.
Why the wait queue matters
If you enable indexer acknowledgment, the forwarder uses a wait queue to
manage the acknowledgment process. Because the forwarder sends data blocks
continuously and does not wait for acknowledgment before sending the next
block, its wait queue will typically maintain many blocks, each waiting for its
acknowledgment. The forwarder will continue to send blocks until its wait queue
is full, at which point it will stop forwarding. The forwarder then waits until it
receives an acknowledgment, which allows it to release a block from its queue
and thus resume forwarding.
A wait queue can fill up when something is wrong with the network or indexer;
however, it can also fill up even when the indexer is functioning normally. This is
because the indexer only sends the acknowledgment after it has written the data
to the file system. Any delay in writing to the file system will slow the pace of
acknowledgment, leading to a full wait queue.
There are a few reasons that a normal functioning indexer might delay writing
data to the file system (and so delay its sending of acknowledgments):
The indexer is very busy. For example, at the time the data arrives, the
indexer might be dealing with multiple search requests or with data
coming from a large number of forwarders.
The indexer is receiving too little data. For efficiency, an indexer only
writes to the file system periodically -- either when a write queue fills up or
after a timeout of a few seconds. If a write queue is slow to fill up, the
indexer will wait until the timeout to write. If data is coming from only a few
forwarders, the indexer can end up in the timeout condition, even if each
of those forwarders is sending a normal quantity of data. Since write
queues exist on a per hot bucket basis, the condition occurs when some
particular bucket is getting a small amount of data. Usually this means that
a particular index is getting a small amount of data.
117
To ensure that throughput does not degrade because the forwarder is waiting on
the indexer for acknowledgment, you should ordinarily retain the default setting of
maxQueueSize=auto. In rare cases, you might need to increase the wait queue
size, so that the forwarder has sufficient space to maintain all blocks in memory
while waiting for acknowledgments to arrive. On the other hand, if you have
many forwarders feeding a single indexer and a moderate number of data
sources per forwarder, you might be able to conserve a few megabytes of
memory by using a smaller size.
118
This topic describes how to filter and route event data to Splunk Enterprise
instances. See "Forward data to third-party systems" in this manual for
119
Configure routing
This is the basic pattern for defining most routing scenarios (using a heavy
forwarder):
1. Determine what criteria to use for routing. How will you identify categories of
events, and where will you route them?
2. Edit props.conf to add a TRANSFORMS-routing attribute to determine routing
based on event metadata:
[<spec>]
TRANSFORMS-routing=<transforms_stanza_name>
[<transforms_stanza_name>]
REGEX=<routing_criteria>
120
DEST_KEY=_TCP_ROUTING
FORMAT=<target_group>,<target_group>,....
Note:
<transforms_stanza_name> must match the name you defined in
props.conf.
In <routing_criteria>, enter the regex rules that determine which events
get routed. This line is required. Use "REGEX = ." if you don't need
additional filtering beyond the metadata specified in props.conf.
DEST_KEY should be set to _TCP_ROUTING to send events via TCP. It can
also be set to _SYSLOG_ROUTING or _HTTPOUT_ROUTING for other output
processors.
Set FORMAT to a <target_group> that matches a target group name you
defined in outputs.conf. A comma separated list will clone events to
multiple target groups.
Examples later in this topic show how to use this syntax.
4. Edit outputs.conf to define the target group(s) for the routed data:
[tcpout:<target_group>]
server=<ip>:<port>
Note:
Set <target_group> to match the name you specified in transforms.conf.
Set the IP address and port to match the receiving server.
The use cases described in this topic generally follow this pattern.
[default]
TRANSFORMS-routing=errorRouting
[syslog]
TRANSFORMS-routing=syslogRouting
2. Edit transforms.conf to set the routing rules for each routing transform:
[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup
[syslogRouting]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=syslogGroup
Note: In this example, if a syslog event contains the word "error", it will route to
syslogGroup, not errorGroup. This is due to the settings previously specified in
props.conf. Those settings dictated that all syslog events be filtered through the
syslogRouting transform, while all non-syslog (default) events be filtered through
the errorRouting transform. Therefore, only non-syslog events get inspected for
errors.
3. Edit outputs.conf to define the target groups:
[tcpout]
defaultGroup=everythingElseGroup
[tcpout:syslogGroup]
server=10.1.1.197:9996, 10.1.1.198:9997
[tcpout:errorGroup]
server=10.1.1.200:9999
[tcpout:everythingElseGroup]
server=10.1.1.250:6666
everythingElseGroup.
[syslog]
TRANSFORMS-routing = routeAll, routeSubset
2. Edit transforms.conf:
[routeAll]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=Everything
[routeSubset]
REGEX=(SYSTEM|CONFIG|THREAT)
DEST_KEY=_TCP_ROUTING
FORMAT=Subsidiary,Everything
3. Edit outputs.conf:
[tcpout]
defaultGroup=nothing
[tcpout:Everything]
disabled=false
server=10.1.12.1:9997
[tcpout:Subsidiary]
disabled=false
sendCookedData=false
server=10.1.12.2:1234
123
For more information, see "Forward data to third party systems" in this manual.
[source::/var/log/messages]
TRANSFORMS-null= setnull
[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue
124
Note: In this example, the order of the transforms in props.conf matters. The null
queue transform must come first; if it comes later, it will invalidate the previous
transform and route all events to the null queue.
1. In props.conf:
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing
2. In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue
[WinEventLog:Security]
TRANSFORMS-wmi=wminull
[wminull]
125
REGEX=(?m)^EventCode=(592|593)
DEST_KEY=queue
FORMAT=nullQueue
[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = _audit
these attributes:
[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal)
Note: The default behavior for heavy forwarders and full Splunk Enterprise
instances changed in Splunk Enterprise version 5.0.2. In earlier versions, the
_internal index was not forwarded by default. Those forwarder types had the
same behavior as the universal forwarder: only data for the _audit internal index
was forwarded.
In most deployments, you will not need to override the default settings. See
outputs.conf for more information on how to whitelist and blacklist indexes. For
more information on default and custom outputs.conf files and their locations,
see "Types of outputs.conf files".
Forward all external and internal index data
If you want to forward all internal index data, as well as all external data, you can
override the default forwardedindex filter attributes like this:
#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =
[tcpout]
#Disable the current filters from the defaults outputs.conf
forwardedindex.0.whitelist =
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =
#Forward data for the "myindex" index
127
forwardedindex.0.whitelist = myindex
This first disables all filters from the default outputs.conf file. It then sets the
filter for your own index. Be sure to start the filter numbering with 0:
forwardedindex.0.
Note: If you've set other filters in another copy of outputs.conf on your system,
you must disable those as well.
You can use the CLI btools command to ensure that there aren't any other filters
located in other outputs.conf files on your system:
This command returns the content of the tcpout stanza, after all versions of the
configuration file have been combined.
Use the forwardedindex attributes with local indexing
On a heavy forwarder, you can index locally. To do that, you must set
indexAndForward attribute to "true". Otherwise, the forwarder just forwards your
data and does not save it on the forwarder. On the other hand, the
forwardedindex attributes only filter forwarded data; they do not filter any data
that gets saved to the local index.
In a nutshell, local indexing and forwarder filtering are entirely separate
operations, which do not coordinate with each other. This can have unexpected
implications when you're performing blacklist filtering:
If you set indexAndForward to "true" and then filter out some data through
forwardedindex blacklist attributes, the blacklisted data will not get
forwarded, but it will still get locally indexed.
If you set indexAndForward to "false" (no local indexing) and then filter out
some data, the filtered data will get dropped entirely, since it doesn't get
forwarded and it doesn't get saved (indexed) on the forwarder.
128
[tcpout:systemGroup]
server=server1:9997
[tcpout:applicationGroup]
server=server2:9997
[monitor://.../file1.log]
_TCP_ROUTING = systemGroup
[monitor://.../file2.log]
_TCP_ROUTING = applicationGroup
The forwarder will route data from file1.log to server1 and data from file2.log
to server2.
1. In outputs.conf:
a. Add the [indexAndForward] stanza:
[indexAndForward]
index=true
selectiveIndexing=true
[tcpout:<target_group>]
server = <ip address>:<port>, <ip address>:<port>, ...
...
[input_stanza]
_INDEX_AND_FORWARD_ROUTING=<any_string>
...
130
[input_stanza]
_TCP_ROUTING=<target_group>
...
[tcpout]
defaultGroup=noforward
disabled=false
[indexAndForward]
index=true
selectiveIndexing=true
[tcpout:indexerB_9997]
server = indexerB:9997
[tcpout:indexerC_9997]
server = indexerC:9997
[monitor:///mydata/source1.log]
_INDEX_AND_FORWARD_ROUTING=local
[monitor:///mydata/source2.log]
_TCP_ROUTING=indexerB_9997
131
[monitor:///mydata/source3.log]
_TCP_ROUTING=indexerC_9997
[tcpout]
defaultGroup=noforward
disabled=false
[indexAndForward]
index=true
selectiveIndexing=true
[tcpout:indexerB_9997]
server = indexerB:9997
[tcpout:indexerC_9997]
server = indexerC:9997
[monitor:///mydata/source1.log]
_INDEX_AND_FORWARD_ROUTING=local
_TCP_ROUTING=indexerB_9997
[monitor:///mydata/source2.log]
_TCP_ROUTING=indexerB_9997
[monitor:///mydata/source3.log]
132
_TCP_ROUTING=indexerC_9997
The only difference from the previous example is that here, you've specified the
_TCP_ROUTING attribute for the input that you're indexing locally. The forwarder will
route both source1.log and source2.log to the indexerB_9997 target group, but it
will only locally index the data from source1.log.
Another way to index one input locally and then forward all inputs
You can achieve the same result as in the previous example by setting the
defaultGroup to a real target group.
1. In outputs.conf, create these stanzas:
[tcpout]
defaultGroup=indexerB_9997
disabled=false
[indexAndForward]
index=true
selectiveIndexing=true
[tcpout:indexerB_9997]
server = indexerB:9997
[tcpout:indexerC_9997]
server = indexerC:9997
[monitor:///mydata/source1.log]
_INDEX_AND_FORWARD_ROUTING=local
[monitor:///mydata/source2.log]
_TCP_ROUTING=indexerB_9997
[monitor:///mydata/source3.log]
_TCP_ROUTING=indexerC_9997
Even though you haven't set up an explicit routing for source1.log, the forwarder
will still forward it to the indexerB_9997 target group, since outputs.conf
specifies that group as the defaultGroup.
133
[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal
_INDEX_AND_FORWARD_ROUTING=local
Steps to deployment
Once you have downloaded the universal forwarder and have planned your
deployment, perform these steps:
1. Install the universal forwarder on a test machine. See below.
2. Perform any post-installation configuration, as described below, here.
3. Test and tune the deployment, as described below.
4. Install the universal forwarder with the tested configuration onto a source
machine.
5. Stop the universal forwarder.
6. Run this CLI command on the forwarder:
./splunk clone-prep-clear-config
This clears instance-specific information, such as the server name and GUID,
from the forwarder. This information will then be configured on each cloned
forwarder at initial start-up.
7. Prep your image or virtual machine, as necessary, for cloning.
8. On *nix systems, set the splunkd daemon to start on boot using cron or your
scheduling system of choice. On Windows, set the service to Automatic but do
not start it.
9. Distribute system image or virtual machine clones to machines across your
environment and start them.
10. Confirm that forwarders have connected to the indexers you specified during
forwarder setup.
135
Referenced procedures
Steps in the above deployment procedure reference these subtopics.
Install the universal forwarder
Install the universal forwarder using the procedure specific to your operating
system:
To install on a *nix machine, see "Deploy a nix universal forwarder
manually".
For a Windows machine, you can use the installer GUI or the command
line interface. To install with the GUI, see "Deploy a Windows universal
forwarder via the installer GUI". For information on the command line
interface, see "Deploy a Windows universal forwarder via the command
line".
Important: On a Windows machine, if you do not want the universal forwarder to
start immediately after installation, you must use the command line interface.
Using the proper command line flags, you can configure the universal forwarder
so that it does not start on the source machine when installed but does start
automatically on the clones, once they're activated.
At the time of installation, you can also configure the universal forwarder. See
"General configuration issues" in the Deployment Overview.
Perform additional configuration
You can update your universal forwarder's configuration, post-installation, by
directly editing its configuration files, such as inputs.conf and outputs.conf.
See "Configure the universal forwarder" for information.
For information on distributing configuration changes across multiple universal
forwarders, see "About deployment server" in the Updating Splunk Enterprise
Instances manual.
Test the deployment
Test your configured universal forwarder on a single machine, to make sure it
functions correctly, before deploying the universal forwarder across your
environment. When testing the deployment, ask these questions:
136
1. Do the data inputs that you configured in the forwarder collect the data
you want?
If they don't:
Check the inputs.conf on the forwarder and confirm that the input stanzas
are correct. For example, if you want to configure monitoring a file, confirm
that the inputs.conf on the forwarder references that file.
Confirm that the stanza that references the file is not disabled (look for
'disabled = 1' in the stanza.)
2. Does the forwarder send the data you expect to the place you expect it?
If it doesn't:
Confirm that the outputs.conf on the forwarder has been correctly
configured. The outputs.conf file should reference a receiving indexer that
the forwarder can access over the network via a host name or IP address
and port that you specify.
Confirm that no firewall blocks network traffic on the ports you specify on
both the forwarder and receiver.
Confirm that the ports you specify on the forwarder and receiver are the
same, as they must be for forwarding to occur. For example, if you specify
port 9997 as the receiving port on the indexer, you must specify this same
port as the target in the outputs.conf configuration on the forwarder.
Use the Search page on the receiving indexer to confirm that you see
events that you configured on the forwarder.
137
TCP data
To forward TCP data to a third-party system, edit the forwarder's outputs.conf file
to specify the receiving server and port. You must also configure the receiving
server to expect the incoming data stream on that port. You can use any kind of
forwarder, such as a universal forwarder, to perform this type of forwarding.
To route the data, you need to use a heavy forwarder, which has the ability to
parse data. Edit the forwarder's props.conf and transforms.conf files as well as
outputs.conf.
Edit the configuration files
To simply forward data, edit outputs.conf:
Specify target groups for the receiving servers.
Specify the IP address and TCP port for each receiving server.
Set sendCookedData to false, so that the forwarder sends raw data.
To route and filter the data (heavy forwarders only), also edit props.conf and
transforms.conf:
In props.conf, specify the host, source, or sourcetype of your data stream.
Specify a transform to perform on the input.
In transforms.conf, define the transform and specify _TCP_ROUTING. You
can also use regex to further filter the data.
Forward all data
This example shows how to send all the data from a universal forwarder to a
third-party system. Since you are sending all the data, you only need to edit
outputs.conf:
[tcpout]
[tcpout:fastlane]
server = 10.1.1.35:6996
sendCookedData = false
138
[host::nyc*]
TRANSFORMS-nyc = bigmoney
[bigmoney]
REGEX = .
DEST_KEY=_TCP_ROUTING
FORMAT=bigmoneyreader
[tcpout]
defaultGroup = default-clone-group-192_168_1_104_9997
[tcpout:default-clone-group-192_168_1_104_9997]
server = 192.168.1.104:9997
[tcpout:bigmoneyreader]
server=10.1.1.197:7999
sendCookedData=false
The forwarder will send all data from host names beginning with nyc to the
non-Splunk server specified in the bigmoneyreader target group. It will send data
from all other hosts to the server specified in the
default-clone-group-192_168_1_104_9997 target group.
Note: If you want to forward only the data specifically identified in props.conf
and transforms.conf, set defaultGroup=nothing.
139
Syslog data
You can configure a heavy forwarder to send data in standard syslog format. The
forwarder sends the data through a separate output processor. You can also filter
the data with props.conf and transforms.conf. You'll need to specify
_SYSLOG_ROUTING as the DEST_KEY.
Note: The syslog output processor is not available for universal or light
forwarders.
The syslog output processor sends RFC 3164 compliant events to a
TCP/UDP-based server and port, making the payload of any non-compliant data
RFC 3164 compliant. Yes, that means Windows event logs!
To forward syslog data, identify the third-party receiving server and specify it in a
syslog target group in the forwarder's outputs.conf file.
Note: If you have defined multiple event types for syslog data, the event type
names must all include the string "syslog".
Forward syslog data
In outputs.conf, specify the syslog target group:
[syslog:<target_group>]
<attribute1> = <val1>
<attribute2> = <val2>
...
Value
This must be in the format
<ipaddress_or_servername>:<port>.
server
n/a
This is a
combination of the IP address or servername of the syslog
server and the port on which the syslog server is listening.
Note that syslog servers use port 514 by default.
140
Optional
Attribute
type
priority
Default
udp
<13> this
signifies
a facility
of 1
("user")
and a
severity
of 5
("notice")
Value
The transport protocol. Must be set to "tcp" or
"udp".
Syslog priority. This must be an integer 1 to 3
digits in length, surrounded by angle brackets;
for example: <34>. This value will appear in the
syslog header.
Mimics the number passed via syslog interface
call; see outputs.conf for more information.
Compute the priority value as (<facility> * 8) +
<severity>. If facility is 4 (security/authorization
messages) and severity is 2 (critical conditions),
priority value will be: (4 * 8) + 2 = 34, which you
specify in the conf file as <34>.
syslogSourceType n/a
timestampformat
""
[host::nyc*]
TRANSFORMS-nyc = send_to_syslog
141
[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group
[syslog:my_syslog_group]
server = loghost.example.com:514
142
Default versions
Splunk Enterprise ships with these default versions of outputs.conf:
On the universal forwarder: The universal forwarder has two default
outputs.conf files, one in $SPLUNK_HOME/etc/system/default and the
other in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default. The
default version in the SplunkUniversalForwarder app has precedence
over the version under /etc/system/default.
On heavy and light forwarders: These have a single default
outputs.conf file, located in $SPLUNK_HOME/etc/system/default.
Important: Do not touch default versions of any configuration files, for reasons
explained in "About configuration files".
Custom versions
When you configure forwarding behavior, those changes get saved in custom
versions of outputs.conf. There are several ways you can specify forwarding
behavior:
While installing the forwarder (Windows universal forwarder only)
By running CLI commands
By using Splunk Web (heavy/light forwarders only)
By directly editing an outputs.conf file
The forwarder automatically creates or edits custom versions of outputs.conf in
response to the first three methods. The locations of those versions vary,
depending on the type of forwarder and other factors:
The universal forwarder. If you use the CLI to make changes to
universal forwarder output behavior, it creates or edits a copy of
outputs.conf in $SPLUNK_HOME/etc/system/local. However, the Windows
installation process writes configuration changes to an outputs.conf file
located in the MSICreated app. For more information on configuring the
universal forwarder, see "Configure the universal forwarder."
Heavy and light forwarders. When you enable a heavy/light forwarder
through Splunk Web or the CLI, an outputs.conf file gets created in the
directory of the currently running app. For example, if you're working in the
search app, the file is created in $SPLUNK_HOME/etc/apps/search/local/.
You can then edit it there.
143
In addition to any outputs.conf files that you create and edit indirectly (for
example, through the CLI), you can also create or edit an outputs.conf file
directly. It is recommended that you work with just a single copy of the file, which
you place in $SPLUNK_HOME/etc/system/local/. (If a copy of the file already
exists in that directory, because of configuration changes made through the CLI,
just edit that copy.) For purposes of distribution and management simplicity, you
can combine settings from all non-default versions into a single custom
outputs.conf file.
After making changes to outputs.conf, you must restart the forwarder for the
changes to take effect.
For detailed information on outputs.conf, look here for the spec and examples.
Configuration levels
There are two types of output processors: tcpout and syslog. You can configure
them at three levels of stanzas:
Global. At the global level, you specify any attributes that you want to
apply globally, as well as certain attributes only configurable at the
system-wide level for the output processor. This stanza is optional.
Target group. A target group defines settings for one or more receiving
indexers. There can be multiple target groups per output processor. Most
configuration settings can be specified at the target group level.
Single server. You can specify configuration values for single servers
(receivers) within a target group. This stanza type is optional.
Configurations at the more specific level take precedence. For example, if you
specify compressed=true for a target group, the forwarder will send the servers in
that target group compressed data, even if compressed is set to "false" for the
global level.
Note: This discussion focuses on the tcpout processor, which uses the [tcpout]
header. For the syslog output processor, see "Forward data to third-party
systems" for details.
Global stanza
Here you set any attributes that you want to apply globally. This stanza is not
required. However, there are several attributes that you can set only at the global
level, including defaultGroup and indexAndForward.
144
The global stanza for the tcpout procesor is specified with the [tcpout] header.
Here's an example of a global tcpout stanza:
[tcpout]
defaultGroup=indexer1
indexAndForward=true
To set default groups for automatic forwarding, include the defaultGroup attribute
at the global level, in your [tcpout] stanza:
[tcpout]
defaultGroup= <target_group1>, <target_group2>, ...
145
[tcpout:<target_group>]
server=<receiving_server1>, <receiving_server2>, ...
<attribute1> = <val1>
<attribute2> = <val2>
...
[tcpout-server://<ipaddress_or_servername>:<port>]
<attribute1> = <val1>
<attribute2> = <val2>
...
Example
The following outputs.conf example contains three stanzas for sending tcpout to
Splunk Enterprise receivers:
Global settings. In this example, there is one setting, to specify a
defaultGroup.
Settings for a single target group consisting of two receivers. Here, we are
specifying a load-balanced target group consisting of two receivers.
Settings for one receiver within the target group. In this stanza, you can
specify any settings specific to the mysplunk_indexer1 receiver.
146
[tcpout]
defaultGroup=my_indexers
[tcpout:my_indexers]
server=mysplunk_indexer1:9997, mysplunk_indexer2:9996
[tcpout-server://mysplunk_indexer1:9997]
[tcpout:my_LB_indexers]
server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995
The forwarder will load balance between the three receivers listed. If one receiver
goes down, the forwarder automatically switches to the next one available.
Data cloning
To perform data cloning, specify multiple target groups, each in its own stanza.
In data cloning, the forwarder sends copies of all its events to the receivers in two
or more target groups. Data cloning usually results in similar, but not necessarily
exact, copies of data on the receiving indexers. Here's an example of how you
set up data cloning:
[tcpout]
defaultGroup=indexer1,indexer2
[tcpout:indexer1]
server=10.1.1.197:9997
[tcpout:indexer2]
server=10.1.1.200:9997
147
The forwarder will send duplicate data streams to the servers specified in both
the indexer1 and indexer2 target groups.
Data cloning with load balancing
You can combine load balancing with data cloning. For example:
[tcpout]
defaultGroup=cloned_group1,cloned_group2
[tcpout:cloned_group1]
server=10.10.10.1:9997, 10.10.10.2:9997, 10.10.10.3:9997
[tcpout:cloned_group2]
server=10.1.1.197:9997, 10.1.1.198:9997, 10.1.1.199:9997,
10.1.1.200:9997
The forwarder will send full data streams to both cloned_group1 and
cloned_group2. The data will be load-balanced within each group, rotating
among receivers every 30 seconds (the default frequency).
Note: For syslog and other output types, you must explicitly specify routing as
described here: "Route and filter data".
Default
defaultGroup
n/a
indexAndForward
false
Where
configured
global
stanza
global
stanza
148
Value
A comma-separated list of one or
more target groups. Forwarder
sends all events to all specified
target groups. Don't set this
attribute if you don't want events
automatically forwarded to a target
group.
If set to "true", the forwarder will
index all data locally, in addition to
forwarding the data to a receiving
indexer.
Important: This attribute is only
available for heavy forwarders. A
universal forwarder cannot index
locally.
server
n/a
<ipaddress_or_servername>:<port>,
where <port> is the receiving
disabled
false
any stanza
level
sendCookedData
true
global or
Specifies whether data is cooked
target group
before forwarding.
stanza
false
global or
Specifies whether the forwarder
target group
sends compressed data.
stanza
compressed
ssl....
useACK
n/a
any stanza
level
false
dnsResolutionInterval 300
The outputs.conf.spec file, which you can find here, along with several
examples, provides details for these and all other configuration options. In
addition, most of these settings are discussed in topics dealing with specific
forwarding scenarios.
Note: In 4.2, the persistent queue capability was much improved. It is now a
feature of data inputs and is therefore configured in inputs.conf. It is not related in
any way to the previous, deprecated persistent queue capability, which was
configured through outputs.conf. See "Use persistent queues to prevent data
loss" for details.
DNS resolution interval
The dnsResolutionInterval attribute specifies the base time interval (in
seconds) at which receiver DNS names will be resolved to IP addresses. This
value is used to compute the run-time interval as follows:
150
Troubleshoot forwarding
Troubleshoot forwarder/receiver connection
This topic discusses troubleshooting steps you can take when forwarders do not
forward or receivers do not receive data. It applies to all forwarder types,
including the universal forwarder.
151
Stopping all listening ports. Queues blocked for more than N seconds.
152
Disable receiving
To disable receiving through the CLI, run the splunk disable listen command:
You can also disable receiving by deleting the [splunktcp] stanza from
inputs.conf.
Answers
Have questions? Visit Splunk Answers and see what questions and answers the
Splunk community has around configuring forwarding.
153
[pipeline:distributedSearch]
disabled = true
For a detailed view of the exact configuration, see the configuration files for the
SplunkForwarder application in
$SPLUNK_HOME/etc/apps/SplunkForwarder/default.
154
[pipeline:indexerPipe]
disabled_processors= indexandforward, diskusage,
signing,tcp-output-generic-processor, syslog-output-generic-processor,
http-output-generic-processor, stream-output-processor
[pipeline:distributedDeployment]
disabled = true
[pipeline:distributedSearch]
disabled = true
[pipeline:fifo]
disabled = true
[pipeline:merging]
disabled = true
[pipeline:typing]
disabled = true
[pipeline:udp]
disabled = true
[pipeline:tcp]
disabled = true
[pipeline:syslogfifo]
disabled = true
[pipeline:syslogudp]
disabled = true
[pipeline:parsing]
disabled_processors=utf8, linebreaker, header, sendOut
[pipeline:scheduler]
disabled_processors = LiveSplunks
These modules include the deployment server (not the deployment client),
distributed search, named pipes/FIFOs, direct input from network ports, and the
scheduler.
155
The defaults for the light forwarder can be tuned to meet your needs by
overriding the settings in
$SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/default-mode.conf
on
a case-by-case basis.
Purge old indexes
When you convert an indexer instance to a light forwarder, among other things,
you disable indexing. In addition, you no longer have access to any data
previously indexed on that instance. However, the data still exists.
If you want to purge that data from your system, you must first disable the
SplunkLightForwarder app, then run the CLI clean command, and then renable
the app. For information on the clean command, see "Remove indexed data from
Splunk" in the Managing Indexers and Clusters of Indexers manual.
156