Technical Note

TN1086: Verint SA Platform and

Verint Enterprise VMS under
VMware 6.7
Publication Date: December 06, 2018

Products •  Verint Situational Awareness Platform (Verint SA) 7.5 SP1 and
higher
•  Verint Enterprise VMS 7.5 SP1 and higher
•  VMware 6.7 Update 1 or higher
Exposure Level Customer Facing
Technical Note Type Product Integration
Description This technical note describes the recommendations for
integrating VMware solutions with the Verint Situational
Awareness Platform (Verint SA) and Verint Enterprise VMS
servers in a High Availability deployment, with a reduced IT
footprint. It also offers troubleshooting information for
performance and ready time.

For details on the memory, network and storage requirements for

the Verint Situational Awareness Platform, refer to the Virtual
Machine Requirements section of Verint Situational Awareness (SA)
Customer Furnished Equipment (CFE) Guide. For details on the
memory, network and storage requirements for the Verint
Enterprise VMS servers, refer to the Verint Enterprise VMS
Customer Furnished Equipment (CFE) Guide.

Unauthorized use, duplication, or modification of this document in whole or in part without the written consent of Verint Systems
Inc. is strictly prohibited. By providing this document, Verint Systems Inc. is not making any representations regarding the
correctness or completeness of its contents and reserves the right to alter this document at any time without notice. Features
listed in this document are subject to change. Contact your Verint representative for current product features and specifications.
All marks referenced herein with the ® or TM symbol are registered trademarks or trademarks of Verint Systems Inc. or its
subsidiaries. All rights reserved. All other marks are trademarks of their respective owners.
© 2018 Verint Systems Inc. All Rights Reserved Worldwide.
 Contents

Contents
Contents 2
Summary of Changes 3
December 2018 3
April 2018 3
November 2017 4
September 2017 4
July 2016 4
VMware High Availability (HA) 5
Prerequisites 5
Additional Information 6
Hardware Requirements 6
Storage Requirements 7
Multicast Filtering 7
Multicast Filtering Modes 8
Enabling Multi-Cast Snooping on a vSphere Distributed Switch 8
Troubleshooting 8
Verifying Total CPU Use 10
Monitoring CPU Performance 10
Assessing CPU Ready-Time Status 11
Identifying a High CPU Ready Time 11
Optimizing the CPU Ready Time 14

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 2

 Summary of Changes

Summary of Changes

December 2018
•  Modifications made throughout document to reflect support of VMware 6.7.
•  "Identifying a High CPU Ready Time" (page 11) updated:
•  New task added
•  Ready Time Parameters table was updated.
•  New guidelines were added to "Optimizing the CPU Ready Time" (page 14)

April 2018
•  Verint SA and Enterprise VMS
•  Combining Verint SA and Verint Enterprise VMS information in TN1070, making TN834
obsolete.
•  "Hardware Requirements" (page 6)- Made minor updates to section.
•  "Multicast Filtering Modes" (page 8)- Added a second note.
•  Verint Enterprise VMS
•  Document Title - Changed the title from Verint VMS Master Server under VMware.
•  Virtual Machine Requirements - Moved this section to the Verint Enterprise VMS Customer
Furnished Equipment Guide.
•  Limitations - Moved this section to the Verint Enterprise VMS Customer Furnished
Equipment Guide.
•  Cover Page - Added a reference to the Verint Enterprise VMS Customer Furnished
Equipment Guide for the Virtual Machine requirements.

•  Verint SA
•  Cover Page - Added a reference to the Verint SA Customer Furnished Equipment Guide
for the Virtual Machine requirements.
•  Virtual Machine Requirements - Removed this section.
•  Removing all mention of fault tolerance from document.

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 3

 November 2017

November 2017
•  Support for Vmware 6.5
•  "Prerequisites" (page 5) - Updated software versions.
•  "Additional Information" (page 6) - Hyperlink updated.
•  "Multicast Filtering" (page 7) - New section.

September 2017
•  Virtual Deployment Requirements - Divided existing table into two to differentiate between
standard and high-end configurations. Section was subsequently moved to the Verint
Enterprise VMS Customer Furnished Equipment Guide.

July 2016
•  First Release

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 4

 VMware High Availability (HA)

VMware High Availability (HA)

VMware High Availability (HA) provides easy-to-use, cost effective high availability for
applications running in virtual machines. In the event of physical server failure, affected virtual
machines are automatically restarted on other production servers with spare capacity. In the
case of operating system failure, VMware HA restarts the affected virtual machine on the same
physical server.

VMware HA allows IT organizations to:

•  Minimize unplanned downtime and IT service disruption, while eliminating the need for
dedicated standby hardware and installation of additional software.
•  Provide affordable, uniform high availability across the entire virtualized IT environment,
without the cost and complexity of failover solutions tied to either operating systems or
specific application.

Prerequisites
The environment must meet the following software and system prerequisites:

•  VMware ESXi 6.7 U1

•  VMware vCenter Server 6.7 U1 or VMware vCenter Appliance 6.7 U1
•  Microsoft Windows Server 2012/2012 R2, or Microsoft Windows Server 2016 (guest
operating system running under ESXi)
•  Cluster configuration with a minimum of two ESX hosts running, enabling HA feature.
•  Shared storage (iSCSI or FC) configured and recognized for all ESX hosts within the cluster.

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 5

 Additional Information

Additional Information
For more information about using VMware 6.7 in an High Availability environment, refer to the
VMware website.

•  Performance Best Practices for VMware vSphere 6.7

•  VMware High Availability

Hardware Requirements
It is highly recommended that the servers, storage adaptors, and other hardware you are using in
the integration are officially supported by VMware. For the list of supported hardware, see
http://www.vmware.com/resources/compatibility/search.php.

Note
The recommendations for running Verint Enterprise VMS or Verint SA in a VMware
environment are based on hardware requirements described in the respective Customer
Furnished Equipment and Verint Supplied Equipment guides.

The following table describes the hardware requirements of a system where VMs are running in
a VMware vSphere environment.

Component Specification
CPU Intel “Haswell” generation (such as Intel Xeon 2600-v3 Series) or later,
according to requirements in the respective Customer Furnished
Equipment (CFE) and Verint Supplied Equipment (VSE) Guides.
Memory 6% more than total running VMs
Network •  VM Network (Verint Enterprise VMS/Verint SA LAN Network).
•  VMKernel Network (vMotion).
Storage •  Verint SA - Virtual datastore on iSCSI, SAS or FC supported by
VMware for the VM.
•  Verint Enterprise VMS - Virtual datastore on iSCSI, SAS or FC
supported by VMware for the VM.

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 6

 Storage Requirements

Recommendations:

•  Verint SA - Configure a dedicated vSwitch for each VMKernel network (vMotion), ensure
that the vSwitch is not shared between the VMKernel and Virtual Machine port groups.

•  Verint Enterprise VMS - Configure a dedicated vSwitch for VMKernel (ensure that the
vSwitch is not shared between the VMKernel and Virtual Machine port groups).

Configure a dedicated vSwitch for VMKernel (ensure that the vSwitch is not shared between
the VMKernel and Virtual Machine port groups).

Configure a dedicated vSwitch for each VMKernel network (vMotion ). Ensure that the vSwitch
is not shared between the VMKernel and Virtual Machine port groups.

Storage Requirements
This section describes the available Storage Area Network (SAN) options that can be used with
the Verint Recorder and ESM servers or with Verint SA servers.

•  Fibre Channel SAN: It is best to use a Fibre Channel SAN when ESXi is hosting any number
of Verint Recorder, Verint ESM, or Verint SA servers. The recommended minimum Fibre
Channel speed is 4Gbps or higher for Verint SA and 8Gbps or higher for Verint Enterprise
VMS.
•   iSCSI SAN: A typical legacy iSCSI SAN provides lower performance than a Fiber Channel
SAN because it generally uses a 1Gbps Network Interface Card (NIC). When using an iSCSI
SAN in an HA deployment, it is recommended to use a 10Gbps NIC and enable jumbo
frames.

Recommendations:

•  Use a Fibre Channel SAN whenever possible.

•  For the best storage performance, use VAAI-capable storage hardware.

Multicast Filtering
Starting from vSphere 6.0 and later, vSphere Distributed Switch supports basic and snooping
models for filtering of multicast packets that are related to individual multicast groups. Choose a
model according to the number of multicast groups to which the virtual machines on the switch
subscribe.

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 7

 Multicast Filtering Modes

Multicast Filtering Modes

There are two multicast filtering modes:

•  Basic Multicast Filtering - This is the default mode, which can be potentially mapped up to 32
IP multicast groups, which is equivalent to 32 cameras.
•  Multicast Snooping - This is the advanced mode, supported only by distributed switches that
can receive multicast traffic on a single switch port from up to 256 IP multicast groups, which
is equivalent to 256 cameras.

Note
•  For details about VMware Multicast Filtering, see Multicast Filtering Modes.
•  The Virtual Machine Network Adaptor type must be 'VMXNET3 Network Adaptor'.

Enabling Multi-Cast Snooping on a vSphere

Distributed Switch
To allow the recorder to support 256 cameras, you must enable advanced multi-cast snooping
on the vSphere Distributed Switch.

▼  Enabling Multi-Cast Snooping on a vSphere Distributed Switch

1. In the vSphere Client (HTML5), navigate to the distributed switch.

2. From the Actions menu, select Settings > Edit Settings.
3. In the dialog box that displays the settings of the switch, click Advanced.
4. From the Multicast filtering mode drop-down menu, select IGMP/MLD snooping, and click
OK.

Note
Multicast snooping is active on hosts running ESXi 6.0 and later.

Troubleshooting
This section describes how to identify and fix problems in a noticeably degraded Verint
Enterprise VMS or Verint SA system that uses VMware. It includes procedures for identifying
performance issues due to overloading of the virtual machine’s host.

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 8

 Troubleshooting

A basic understanding of VMware is required in order to perform the troubleshooting tasks in

this section, such as how to use the vSphere Client and the vCenter.

The following troubleshooting information is included:

•  Monitoring performance
•  Assessing CPU ready time

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 9

 Verifying Total CPU Use

Verifying Total CPU Use

It is very important to ensure that the VMs are not overloaded with CPU tasks.

▼  To verify the total CPU use

1. Select the ESX host in the vSphere Client.

2. Select the Performance tab. A graph is displayed showing the real-time activity for CPU,
memory, disk.

3. Verify that the Average Percentage on the host and all the cores does not exceed 75% of the
total CPU.

If the Average Percentage is above 75%, consider adding more ESX hosts to your data
center to balance the load more effectively.

Monitoring CPU Performance
The vSphere Client can be used to identify problems on both the physical host machine and the
VM.

In the vSphere Client application, there are two important symbols to pay attention to, the
Yellow Caution Alert and the Red Alert. These symbols indicate a problem that must be either
resolved or monitored. They appear when a threshold that has been defined in a rule is
surpassed, with or without a condition. Examples include sending a warning if the CPU exceeds
85% for five minutes, and sending an alert if the CPU reaches 100% for five minutes.

▼  To monitor CPU performance

1. Access vSphere by logging into one of the following:

•  vSphere Client (HTML5)

•  vSphere Web Client (Flex)
2. Do one of the following:

•  Highlight the ESX host for general real-time data capture.

•  Highlight any specific VM for more accurate data.
3. Click the Monitor tab, and under Performance, select Overview or Advanced. The Real-Time
Graph is displayed.

10 © 2018 Verint Systems Inc. All Rights Reserved Worldwide.

 For troubleshooting purposes, the Real-Time Graph can be exported as an image, or to a
Microsoft Excel file with all the currently displayed data. This can help track multiple time
slots in order to analyze if CPU or memory use are growing with time.

4. (Optional) Click Chart Options to specify a range of dates, objects and counters to be
displayed on the chart.

For example, choose the option Past Year with Object 1, and Object 2 for CPU Core 1 and 2,
and the counter Usage as an average. Then, export the results for further analysis.

Assessing CPU Ready-Time Status

To ensure that the system is properly installed and configured, you need to confirm that the
CPU ready-time status of the virtual machines is appropriate. CPU ready time is the time that a
VM waits in a ready-to-run state before being scheduled on a CPU. A low ready time is
acceptable and does not noticeably affect performance. The higher the ready time, the more it
will affect VM performance.

This section describes the following:

•  Identifying a High CPU Ready Time

•  Optimizing the CPU Ready Time
•  Verifying Total CPU Use

Identifying a High CPU Ready Time

The CPU ready time can be affected by any of the following:

•  CPU Over-subscription: Maintain a healthy ratio, less than 1:3 between a pCPU (Physical
CPU) to vCPU (Virtual CPU) per each ESX host.
•  CPU Affinity/Limitation: Avoid setting a CPU Affinity and/or CPU Limitation for Virtual
Machines.
•  Large number of VMs compared to hardware availability for overall CPU utilization
A high CPU ready time can cause the following problems:

•  A slow VM
•  Loss of communication with the VM, causing network failure or bad data transfer
When you measure CPU Ready Time through the VSPHERE CLIENT (HTML5) at the ESX host
level, it might be difficult to identify the root cause of latency, therefore it is recommended to
explore the CPU ready time at the virtual machine level.

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 11

 Identifying a High CPU Ready Time

▼  To identify a high CPU ready time at the virtual machine (VM) level

1. Access vSphere by logging in with the vSphere client (HTML5)

2. Highlight a specific VM.
3. Click the Monitor tab, and under Performance, select Advanced.
4. From the View drop down, select CPU ready.

For more information, see Converting between CPU summation and CPU % ready values.

You can also identify a high CPU ready time by using the esxtop command from an ESXi SSH
connection.

▼  To identify a high CPU ready time using the esxtop command from an ESXi SSH connection

1. Log in to the ESX host that contains the VM using an SSH connection.

Note
Ensure that the ESX host is enabled for SSH remote connections.

2. In the console, type esxtop.

3. Press c on the keyboard.

4. Look at the %RDY column for the VMs and the console.

12 © 2018 Verint Systems Inc. All Rights Reserved Worldwide.

Ready Time Parameters Description
Load Average This is the use for all physical CPUs. It displays for one-,
five-, and fifteen-minute intervals
PCPU Percentage of the individual physical CPU
CPU Panel Percentage of total CPU time:

•  %USED - User Time

•  %SYS - System Time
•  %WAIT - Wait Time
•  %RDY - Percentage of time that the resource was
ready to run but was not provided a CPU resource on
which to execute.

For more information please refer to vSphere Monitoring

and Performance.

•  The lower the percentage, the better the CPU ready time. A low percentage indicates
that the virtual machine did not waste time waiting for the next CPU cycle.
•  It is highly recommended you keep VMs below 5% CPU Ready per vCPU.
•  If the percentage is too high, it most probably means that the system is overloaded with
virtual machines, and/or sharing too many resources.
•  Methods for preventing a high CPU ready time include CPU reservation, and setting the
affinity to protect the ESX console from overload. The total CPU use for the host should
be below 75%.

© 2018 Verint Systems Inc. All Rights Reserved Worldwide. 13

 Optimizing the CPU Ready Time

Optimizing the CPU Ready Time

The following guidelines describe some of the ways you can maintain a healthy virtual
infrastructure:

Note
The following are not a comprehensive guide to diagnosing and troubleshooting problems in a
virtual environment

•  Verify that VMware tools are installed on every virtual machine on the host.
•  Monitor and analyze the virtual machine CPU usage.
•  Determine whether there is a high CPU ready time.
•  Increase the shares on a critical virtual machine, entitling the virtual machine to preferential
access to CPU resources.
•  Reduce the number of virtual CPUs on a virtual machine to the number required to execute
the virtual machines' workload.
•  Add more ESX hosts or upgrade the CPUs on the current hosts.

14 © 2018 Verint Systems Inc. All Rights Reserved Worldwide.