AlienVault Data Sources

Adapt collection to your organization

Thursday, March 22, 2012

 Types of DS Connectors
Two types of Data Source Connectors

Detectors: They offer events (Snort, Firewalls, Antivirus, Web

servers, OS events..)

Monitors: They offer indicators (Ntop, Tcptrack, Nmap, Webs,

Compromise & Attack...)

Thursday, March 22, 2012

 Files
Each DS Connector (monitors and detectors) is built on two files:

Plugin.cfg
Contains the configuration parameters of the plugins and the rules
that an event has to match in order to be collected and normalized.

Plugin.sql
Contains the description of every possible event that can be
collected using the plugin (Plugin_id, Plugin_sid, Name given to the
event, priority and reliability)

Thursday, March 22, 2012

 Ds Connector: Detector
[DEFAULT] Numerical identifier of the plugin
plugin_id=4003

# default values for dst_ip and dst_port

# they can be overwritten in each rule
dst_ip=\_CFG(plugin-defaults,sensor)
dst_port=22 Default fields for every event
[config]
type=detector Type of plugin: Detector
enable=yes

source=log
location=/var/log/auth.log
Source of the events (log, mssql,mysql or wmi)
create_file=false

process=sshd
start=no
stop=no
Associated process and start/stop options
startup=/etc/init.d/ssh start
shutdown=/etc/init.d/ssh stop

[ssh - Failed password]

Type of event
# Feb 8 10:09:06 golgotha sshd[24472]: Failed password for dgil from 192.168.6.69
port 33992 ssh2
event_type=event
regexp="(\SYSLOG_DATE)\s+(?P<sensor>[^\s]*).*?ssh.*?Failed password for (?
P<user>\S+)\s+from\s+.*?(?P<src>\IPV4).*?port\s+(?P<sport>\PORT)" Regular expressions
plugin_sid=1
sensor={resolv($sensor)}
date={normalize_date($1)}
src_ip={$src} Fields that will be sent to the AlienVault Server
dst_ip={resolv($sensor)}
src_port={$sport}
username={$user}

Thursday, March 22, 2012

 Ds Connector: Detector
plugin_id

Data Source ID. User reserved range: 9000-10000

E.g.: plugin_id=3000

source

log: Text file (E.g: SSH, Sudo, Apache...)

mssql: Mssql Database (E.g: panda-se)

mysql: Mysql Database (E.g: moodle)

wmi: Windows Management Instrumentation (wmi-system-logger)

Thursday, March 22, 2012

 Ds Connector: Detector
location

- Files in which the applications store the events

- E.g.: location=/var/log/file.log

create_file

- Create the file in case it does not exist

- false/true

process / start / stop / startup / shutdown

- Only if the process is running in the same machine that the detector

- If the process is not running in the machine, is there a process helping

us to collect those logs? syslog? fw1-loggrabber?

Thursday, March 22, 2012

 Ds Connector: Detector
Rules

Rules define the format of each event and how they are normalized

It is composed by a regular expression and the list of fields that the

event will include when once it is sent to the AlienVault SIEM or
Logger

In some cases only one regular expression will collect every event
coming from one application, in some other cases more than one
rule will be required

Thursday, March 22, 2012

 DS Connector: Detector
Rules

Rules are loading in alphabetical order based on the name given to

each rule

Once the log matches one the regex of one rule the ossim agent
stops processing the event

Generic rules must be the last loaded in memory as they will

probably match all the events

The name of the rule is mandatory

Thursday, March 22, 2012

 DS Connector: Detector
The rule must include the event type:

event_type=event

The following fields can be used to normalize the event:

plugin_id plugin_sid date sensor interface protocol

src_ip src_port dst_ip dst_port username password

filename userdata1 userdata2 userdata3 userdata4 userdata5

userdata6 userdata7 userdata8 userdata9

Values in bold are mandatory

Fields in red include values that always have to be defined in the plugin

Fields in green can will be filled by the AlienVault Agent in case they can not be found
in the original log (Dont include that line when creating the plugin)

Fields in grey are optional

Thursday, March 22, 2012

 DS Connector: Detector
Regexp

The regexp field contains the regular expression that defines the format of the events,
and extracts the information to normalize the event.

regexp="(\SYSLOG_DATE)\s+(?P<sensor>[^\s]*).*?ssh.*?Failed password for (?P<user>\S+)\s+.*?(?P<src>\IPV4).*?port\s+(?P<sport>\PORT)"

regexp=(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d)\S+ (\S+) (\S+) (\S+) (\d+) (\w+) (\S+) \S+ (\d+)

The regular expressions are written using the Python regular expression syntax:

http://docs.python.org/library/re.html

10

Thursday, March 22, 2012

 Regular expressions

Operator Meaning
c A non special character matches with itself
Removes the special meaning of the character c; The RE \$ matches with
\c
$
^ Indicates located at the beginning of the line
$ Indicates located at the end of the line
. Any individual character
[] One or any of the characters ; accepts intervals of the type a-z, 0-9, A-Z
[^] A char different from ; Accepts intervals of the type a-z, 0-9, A-Z

11

Thursday, March 22, 2012

 Regular expressions
Regular expression Matches with
a.b axb aab abb aSb a#b ...
a..b axxb aaab abbb a4$b ...
[abc] a b c (one character srtings)
[aA] a A (one character srtings)
[aA][bB] ab Ab aB AB (two character srtings)
[0123456789] 0123456789
[0-9] 0123456789
[A-Za-z] A B C ... Z a b c ... Z
[0-9][0-9][0-9] 000 001 .. 009 010 .. 019 100 .. 999

[0-9]* empty_chain 0 1 9 00 99 123 456 999 9999 ...

0 1 9 00 99 123 456 999 9999 99999

[0-9][0-9]*
99999999 ...
^.*$ A full line

12

Thursday, March 22, 2012

 Regular expressions
Operator Meaning
r* 0 or more occurrences of the RE r
r+ 1 or more occurrences of the RE r
r? 0 or an occurrences of the RE r, and no more
r{n} No occurrences of the RE r
r{,m} 0 or at most m occurrences of the RE r
r{n,m} N or more occurrences of the RE r, but at most m
r1|r2 The RE r1 or the RE r2

Regular Matches with

expression
[0-9]+ 0 1 9 00 99 123 456 999 9999 99999 99999999 ..
[0-9]? empty_string 0 1 2 .. 9
(ab)* empty_string ab ababab abababababab
empty_string 1234ab 9ab9ab9ab 9876543210ab
([0-9]+ab)*
99ab99ab ...

13

Thursday, March 22, 2012

 Regular expressions

Regular Matches with Equals

expression
\d Any decimal character [0-9]
\D Any non decimal character [^0-9]
\s Any space character [ \t\n\r\f\v]
\S Any non space character [^ \t\n\r\f\v]
Any alphanumeric character
\w [a-zA-Z0-9_]
and _
\W Any non alphanumeric character [^a-zA-Z0-9_]
\Z End of line

14

Thursday, March 22, 2012

 Regular expressions
Pattern Description
b,c,X,8 Ordinary characters just match themselves exactly. The meta-characters which do not match themselves because they
have special meanings are: . ^ $ * + ? { [ ] \ | ( )

. Matches any single character except newline (\n).

\w Lowercase w matches a "word" character: a letter or digit or under-bar [a-zA-Z0-9_]. It only matches a single word char,
not a whole word.

\W Uppercase w matches any non-word character.

\s Lowercase s matches a single whitespace character -- space, newline, return, tab, form [ \n\r\t\f].
\S Upper case s matches any non-whitespace character.
\d Lowercase d matches a single Decimal digit [0-9]
\D Uppercase d matches any non decimal character
\t Matches a tab character
\n Matches a newline character
\r Matches a return character
\Z Matches only at the end of the string.
\ Escapes special characters. If you are unsure if a character has special meaning, such as '@', you can put a slash in front
of it, \@, to make sure it is treated just as a character.

15

Thursday, March 22, 2012

 Regex aliases
/etc/ossim/agent/aliases.cfg

/etc/ossim/agent/aliases.local (For user custom aliases)

This file contains predefined regular expressions that can be used

to simplify the process of writing new plugins

Usage Example:

\SYSLOG_DATE\s+\IPV4\s+\IPV4

16

Thursday, March 22, 2012

 Regular Expressions
The information extracted by the regular expression from the log
can be accessed by:

Position: (\d\d):(\d\d):(\d\d)

- hour={$1}

- minutes ={$2}

- seconds={$3}

Tags: (?P<hour>\d\d):(?P<minutes>\d\d)(?P<seconds>\d\d)

- hour={$hour}

- minutes ={$minutes}

- seconds={$seconds}

17

Thursday, March 22, 2012

 Functions
The AlienVault SIEM and Logger must receive normalized events, as an
example the addresses have to use IPV4 format and the date has to use
the following format YYYY-MM-DD HH:MM:SS (2010-12-31 22:57:00)

To simplify the process of normalizing events some functions can be

used

resolv()

Translate hostnames into IPV4 addresess (DNS queries)

normalize_date()

The normalize_date function translate many format dates into the format
accepted by the SIEM or Logger

- YYYY-MM-DD hh:mm:ss

More functions can be found and defined in ParserUtils.py

18

Thursday, March 22, 2012

 Translation Tables
Translations can be configured to be done once the event has
been collected

E.g.: When the event id is not numeric, but plugin_sid has to be

numeric

Translations have to be defined inside a category called

[translation]

Translate using the function translate().

Even more info can be found here:

http://www.alienvault.com/docs/
AlienVault%20Building%20Collector
19 %20Plugins.pdf

Thursday, March 22, 2012

 Hands-On: plugins

firewall logs are sent to /var/log/firewall.log

while truedo cat /var/log/firewall.log | logger -t <STRING> sleep 10done

20

Thursday, March 22, 2012

 Hands-On: plugins
write a firewall plugin

copy existing similar plugin: <plugin.cfg>

plugin_id number start on custom range

write your regex rule to match the loglines

write a firewall sql file

copy existing similar sql file: <plugin.sql>

change the fields to your custom plugin rules

activate your plugin on the CLI (alienvault-setup)

write your sql file to the database

21

Thursday, March 22, 2012