Shedding Light on Side Tables

Stefan Zier
Platform Team
September 2009

2009 ArcSight, Inc. All rights reserved.

ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners.
1

Agenda

Side Tables
What

are they?

What

data is stored?

How

does the Side Table cache work?

What

www.arcsight.com

data causes problems; how do you fix them?

2009 ArcSight Confidential

What are Side Tables?

2009 ArcSight, Inc. All rights reserved.

ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners.
www.arcsight.com

2009 ArcSight Confidential

What Are Side Tables?

What Are Side Tables?

ArcSight
Event
Side

ESM uses Oracle to store all events

schema was expanded with ArcSight 3.0

Tables were introduced in ArcSight 3.0

Normalization

of schema to get big benefits

Avoid repetition and save disk space

Make event insertion and queries much faster
Allow us to have a lot more fields in events

www.arcsight.com

2009 ArcSight Confidential

What Are Side Tables?

How Does Normalization Work?

Side

Tables store parts of an event

Devices
Agents (connectors)
Source/target IP Geo information
Sets of event categories
Sets of additional data labels
References to annotations

All

of these parts have the following properties

Values repeated in many events

Limited number of different values
At most tens of thousands of each

Normalization

www.arcsight.com

avoids repeating values

2009 ArcSight Confidential

What Are Side Tables?

Normalization
Unique Fields

Repeated Fields

EVENT1

NAME

SRC IP

TARGET IP

AGENT ID AGENT NAME

AGENT LOCATION

EVENT2

NAME

SRC IP

TARGET IP

AGENT ID AGENT NAME

AGENT LOCATION

EVENT3

NAME

SRC IP

TARGET IP

AGENT ID AGENT NAME

AGENT LOCATION

EVENT4

NAME

SRC IP

TARGET IP

AGENT ID AGENT NAME

AGENT LOCATION

EVENT5

NAME

SRC IP

TARGET IP

AGENT ID AGENT NAME

AGENT LOCATION

EVENT6

NAME

SRC IP

TARGET IP

AGENT ID AGENT NAME

AGENT LOCATION

!
www.arcsight.com

Repetitive information is saved once

and referenced to its original source

2009 ArcSight Confidential

What Are Side Tables?

Normalized Event Table Schema

arc_event_device
DVC_DESCRIPTOR_ID
DVC_EXTERNAL_ID
DVC_VENDOR
DVC_PRODUCT
DVC_VERSION
DVC_FACILITY
DVC_DNS_DOMAIN
DVC_NT_DOMAIN
DVC_TIME_ZONE
DVC_HOST_NAME
DVC_ADDRESS
DVC_TRANS_ADDRESS
DVC_MAC_ADDRESS DVC_ZONE
DVC_TRANS_ZONE
DVC_ASSET_ID
DVC_INBOUND_INTERFACE
DVC_OUTBOUND_INTERFACE
DVC_PROCESS_NAME

arc_event_geo_location
GEO_DESCRIPTOR_ID
GEO_LATITUDE
GEO_LONGITUDE
GEO_POSTAL_CODE
GEO_COUNTRY_CODE
GEO_REGION_CODE
GEO_LOCATION_INFO
www.arcsight.com

arc_event_label
arc_event
LBL_DESCRIPTOR_ID
DVC_DESCRIPTOR_ID
AGT_DESCRIPTOR_ID
SRC_GEO_ID
DEST_GEO_ID
CAT_DESCRIPTOR_ID

arc_resource_ref
REFERENCE_ID
ID
URI
EXTERNAL_ID
REFERENCE_TYPE

arc_event_category
CAT_DESCRIPTOR_ID
CAT_OBJECT
CAT_BEHAVIOR
CAT_TECHNIQUE
CAT_SIGNIFICANCE
CAT_OUTCOME
CAT_DEVICE_GROUP
CAT_TUPLE_DESCRIPTION
CAT_CUSTOM_FORMAT_FIELD

LBL_DESCRIPTOR_ID
LBL_STRING1_LABEL
LBL_STRING2_LABEL
LBL_STRING3_LABEL
LBL_STRING4_LABEL
LBL_STRING5_LABEL
LBL_STRING6_LABEL
LBL_NUMBER1_LABEL
LBL_NUMBER2_LABEL
LBL_NUMBER3_LABEL
LBL_DATE1_LABEL
LBL_DATE2_LABEL

arc_event_agent
AGT_DESCRIPTOR_ID
AGT_ID
AGT_TYPE
AGT_VERSION
AGT_TIME_ZONE
AGT_HOST_NAME
AGT_ADDRESS
AGT_TRANS_ADDRESS
AGT_MAC_ADDRESS
AGT_ZONE
AGT_TRANS_ZONE
AGT_ASSET_ID
AGT_DNS_DOMAIN
AGT_NT_DOMAIN

2009 ArcSight Confidential

Side Table Caches

2009 ArcSight, Inc. All rights reserved.

ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners.
www.arcsight.com

2009 ArcSight Confidential

Side Table Cache

Side Table Cache

Connectors and consoles deal with denormalized

eventsdo not keep up-to-date with Side Tables

Manager needs to map denormalized events to the

normalized schema in the DB
Read event: Find all values for columns that are in Side Tables
Insert event: Find the ID that matches entry with values

Each operation requires a database query

6-8 database operations per event

Side Table cache is an in-memory cache

Cuts number of operations down per event to 1

www.arcsight.com

2009 ArcSight Confidential

Side Table Cache

Side Table Cache Sizing

Side Table Cache attempts to keep 100%
of entries in memory
Memory is limited, so Side Table caches need a limit
Typically, the size is larger than the number of entries
Size can be reconfigured
Defaults

50,000 Geo Descriptors

2,500 Connectors
50,000 Devices
10,000 Categories
2,000 Labels
20,000 Resource References

www.arcsight.com

2009 ArcSight Confidential

10

Side Table Cache

Side Table Cache Specifics

Two

cache modes

Completeall entries fit into memory

Partialmore entries than memory space

Event

insert

Check whether entry is in cache (memory operation)

In partial mode, try to find entry in database (database operation)
If entry not found, create entry (database operation)

Event

read

Check cache for entry (memory operation)

In partial mode, try to find entry in database (database operation)

www.arcsight.com

2009 ArcSight Confidential

11

Problematic Data

2009 ArcSight, Inc. All rights reserved.

ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners.
www.arcsight.com

2009 ArcSight Confidential

12

Problematic Data

What Data Causes Side Table Issues?

Normalization

and caching assume

limited number of entries

If

number is not limited

Cache will only hold a fraction of entries

For each cache missed, additional database operations
are needed when retrieving and inserting events
Additional database queries multiply the load on the database
Insertions and reads (channels, reports, trends) most likely slow down
Connectors may start to cache
Events take much more space than needed

www.arcsight.com

2009 ArcSight Confidential

13

Problematic Data

How Can This Happen?

Connectors put values into events that are not very repetitive, for
example

Unique event IDs

Timestamps
Checksums
Packet sizes
Strings that contain any of the above

Various reasons
Event format changed on the device
Connector changed
FlexConnector

A single misbehaving connector can affect an entire ArcSight

ESM deployment

www.arcsight.com

2009 ArcSight Confidential

14

Problematic Data

How Can This Happen?

Rules can create additional side table entries
M1 can create a new side table entry
Aggregation on subset of fields in the side table
Each different combination of values creates new entry

Rule of thumb: aggregate on ALL or NONE

of the side table fields

www.arcsight.com

2009 ArcSight Confidential

15

Problematic Data

Countermeasures

ArcSight ESM 4.0 (and earlier versions) implement

countermeasures to prevent the issue
Connectors monitor for non-repetitive values in Side Table fields
Manager monitors rate of new Side Table entries

Notifications are sent when an issue is identified

The manager pauses the misbehaving connector

until an operator fixes the issue

www.arcsight.com

2009 ArcSight Confidential

16

Working With Side Tables

2009 ArcSight, Inc. All rights reserved.

ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners.
www.arcsight.com

2009 ArcSight Confidential

17

Working with Side Tables

Different Notifications

Notifications logged in server.log and as internal events

Side Table cache hit rate low messageearly tell-tale

sign of a runaway Side Table

Agent has been paused messagelate tell-tale sign of

a runaway Side Table

www.arcsight.com

2009 ArcSight Confidential

18

Working with Side Tables

Cache Hit Rate Low Message

If you know that you have an extraordinary number of
devices or other objects (larger than the defaults)

Carefully increase the limit in server.properties

Increasing it to a very large number may run your

manager out of memory

Restart manager

www.arcsight.com

2009 ArcSight Confidential

19

Working with Side Tables

Fixing a Runaway Side Table Problem

Identify the problem

Contact support to get the SideTableStats.sql script
Run it, and support will tell you which column in which
table shows too many distinct values
Run a channel filtering on the connector that was reported
in the message
Look at the values in the column
If you recognize what the data is, this will most likely be your light bulb
moment

Once you have identified it

Adjust your FlexConnector
Or work with support to identify a what can be done
to fix the issue

www.arcsight.com

2009 ArcSight Confidential

20

Summary

Side Tables allow for efficient storage of large event

volumes in Oracle

Side Table caches accelerate normalization on the

manager by orders of magnitude

Misconfigured connectors can cause issues

But both connectors and manager have efficient

countermeasures in place to prevent problems

There are well-established procedures for

troubleshooting Side Table issues

www.arcsight.com

2009 ArcSight Confidential

21