Enterprise Vault Backups

SYMANTEC TECHNOLOGY NETWORK WHITE PAPER: RISK MANAGEMENT

Symantec Technology Network

Enterprise Vault Best Practice

Enterprise Vault Backups

- by -

Steve Blair

Symantec Sr. Technical Product Manager, Enterprise Vault

December 21, 2007

1
Symantec Technology Network
1
 Enterprise Vault Backups

Table of Contents
Table of Contents................................................................................................................................... 2
Overview ................................................................................................................................................... 3
Introduction ........................................................................................................................................ 3
Scope of Document........................................................................................................................... 3
Target Audience................................................................................................................................. 3
Acknowledgements .......................................................................................................................... 3
Backup Trends .................................................................................................................................... 4
Enterprise Vault Infrastructure and your environment ................................................ 4
Backups vs. Archiving ................................................................................................................. 4
Enterprise Vault Backup Requirement Overview ................................................................. 5
High-level Enterprise Vault Deployment .................................................................................................. 7
Enterprise Vault Backup Requirements .................................................................................................... 7
Enterprise Vault Server (1)........................................................................................................... 7
Enterprise Vault Indexes (3)........................................................................................................ 8
Enterprise Vault Shopping Service Locations (3) ................................................................ 8
Enterprise Vault Store Partitions (3) ........................................................................................ 8
SQL Databases (2) ............................................................................................................................ 9
Enterprise Vault Backup Timing and Scheduling.................................................................... 10
Overview of an evening for backups ........................................................................................... 11
Addendum / Additional Information ........................................................................................... 12
PREBACKUP.BAT:............................................................................................................................. 13
POSTBACKUP.BAT: .......................................................................................................................... 13
About the Author....................................................................................................................................... 15
About Symantec Technology Network (STN)......................................................................................... 15

2
Symantec Technology Network
2
 Enterprise Vault Backups

Overview

Introduction

This best practice document discusses the inherent needs to securely backup
operations Enterprise Vault servers to insure best available DR scenarios. With
several very important information sources integrated with Enterprise Vault, it is
critical that proper backups of this data be performed on a regular basis. This paper
considers the impact this has on Enterprise Vault ways customers can easily
implement a high degree of coverage for their archived content. It is important to
understand that it is a Best Practice to regularly backup the components of your
Enterprise Vault environment to insure that you have adequate DR protection as
with any Tier-1 business critical application.

Scope of Document

This document is focused on Enterprise Vault, Microsoft SQL, Vault Store settings,
and the indexes that are part of the environment as a whole. This document
presumes a good working knowledge of your company backup products, schedules,
and DR plans. The implementation, planning and use of actual backup software /
hardware solution are outside the scope of this document. However, Best Practice
recommendations to insure your DR plans protect your data in Enterprise Vault for
use of your backup software are made.

Target Audience

This document is aimed at customers, consultants and support staff and it is

assumed the reader has a good understanding about the architecture and
operational aspects of an Enterprise Vault server, and their internal network and
storage architecture. This document also discusses concepts related to Microsoft
Server management, and expects the reader to either be skilled in this area, or
have team member(s) who are.

Acknowledgements

I would like to acknowledge the contribution that other individuals made towards
making this a successful and informative document. Contributions and feedback
came from the following teams: Technical Product Management, Technical Field
Enablement, Engineering, Consulting, and our Customer Support Teams.

3
Symantec Technology Network
3
 Enterprise Vault Backups

Backup Trends

Companies implementing Enterprise Vault typically purchase to solve one of several

needs. They are either trying to offset the explosive growth of individual mailboxes,
file server personal / shared areas, or insure relevant regulatory compliance in their
country. Backing up your data to insure your company has an adequate safety net
has long been the bane of the modern System Administrator, but lack of backups
place a company at extreme risk in a DR (Disaster Recovery) situation. Because of
implementing Enterprise Vault, a company must undertake a review of their
existing backup schedules and plans, as archiving allows customers more flexibility
and newer ways to have a safety copy.

At some companies they are of a smaller size; thus their Enterprise Vault backup
scenarios are relatively straightforward to deliver. At larger companies, given a
number of Enterprise Vault servers their backup scenarios are slightly more
complicated. Given the advances in both large storage tape technologies such as
DLT^2 and LTO3, and disk systems that operate like tape or replicate, no two
customers studied are performing their backups in a similar fashion.

The one unifying similarity of Enterprise Vault customers regardless of size is that
they are aware they have to insure for their internal &/or external customers a high
degree of data availability.

Enterprise Vault Infrastructure and your environment

As the landscape changes with the introduction of Enterprise Vault into your
environment, it does not absolve one of the responsibilities to insure that adequate
backups are in place. Given the data contained in Enterprise Vault is still just as
valuable, just as important, it is our recommendation just like with email and file
servers, that you treat your Enterprise Vault server(s) as any other Tier-1
application server in your network. Because of installing Enterprise Vault at your
site, you will observe (in many cases) greatly shortened backup times for the
server(s) that Enterprise Vault archives from, but you will still need to make sure
your DR position is strong by the scheduled backups of your
Enterprise Vault data.

Backups vs. Archiving

Let’s begin by reviewing that many people are confused over the two terms
“backup” vs. “archiving” and often wonder if there is any discernable difference.
After all, is a backup not an archive in time, or a backup stored offsite considered
an archive? It is absolutely true that both terms are valid, but the line becomes
clearer when it is time to produce content back from a backup. Backup media is not
the same as a hard disk spinning online, and as such can and does suffer from
media failure, inability to read / restore content, and data loss of the physical
4
Symantec Technology Network
4
 Enterprise Vault Backups

media. The important factor is time to produce / restore and this is where archiving
is unequalled in terms of performance and availability vs. a backup / restore
solution. Active archiving differs from regular backups as it has indexed and
represented the content in a way immediately available for recall, review and use
without operator interaction. The very activity of having the content still stored,
managed, and available replaces many daily System Administrator tasks, and
supplements end-user experience of their data being highly-available via “active
archiving” as delivered with Enterprise Vault. There are two major Best Practices
you have available to perform a backup of your important information stored in
Enterprise Vault: data-only backups of the archived content, and data and
application backups of the whole Enterprise Vault environment. We shall review
both in this paper, and allow you as the reader to take a decision on which will be
the Best Practice for your environment.

Enterprise Vault Backup Requirement Overview

Let us begin by saying that there are several crucial components in your Enterprise
Vault deployment that need to be backed up periodically. While having Enterprise
Vault as your archiving solution greatly reduces your backup reliance, it does not
absolve you of performing backups.

5
Symantec Technology Network
5
 Enterprise Vault Backups

The items in your environment that must be backed up are:

¾ SQL Databases (Directory, Audit, Monitoring, Compliance &/or

Discovery Accelerator, and Vault Stores)
¾ Indexes, their directories and files
¾ Vault Store partition (s) and files
¾ EV Server system for registry settings
¾ EV Server(s) license file (per EV server!)
¾ Any customisations done to your EV environment:
ƒ Scripts
ƒ Welcome Messages
ƒ Webapp.ini
ƒ PST Migration Templates
ƒ Discovery Accelerator and Compliance Accelerator
o Web-config files
o XML templates
o Custom Search word lists

Each of the above components being backed up plays a vital role in the Enterprise
Vault environment as a whole. It is a Best Practice to insure that good backups of
the above environment are performed on a regular basis as part of your total DR
plan. The diagram below shows a sample Enterprise Vault deployment and the
interdependencies used by the various components in a simple deployment. Whilst
this diagram may not accurately reflect the environment you have Enterprise Vault
deployed in at your site, it does cover the important components of the Enterprise
Vault environment.

The theories and Best Practice guidance done in this paper are the results of years
of testing, and actual customer experience to help guide you on making your
environment very reliable. Part of the Best Practice of backing up your Enterprise
Vault environment is to test the backups periodically to insure that your backups
are good. Many customers have relied solely on others or receiving an email “the
backups worked” and never test their backup’s validity.

As with any Tier-1 application, this is a risk to your business that we strongly
encourage you to avoid by performing regular simple checks of your backup
solution. In many cases, customers adhering to these Best Practices have
uncovered that there was a problem with their backup solution only after a
hardware / server failure thus we strongly encourage you to chose a known
industry partner’s products for backups of your Enterprise Vault environment.

6
Symantec Technology Network
6
 Enterprise Vault Backups

High-level Enterprise Vault Deployment

Each item is detailed by number in the next section to fully explain the content and
importance to your implementing Best Practices with backing up your Enterprise
Vault Infrastructure.

Enterprise Vault Backup Requirements

Enterprise Vault Server (1)

Now that we have covered the main components of the archiving infrastructure in
your environment, we have to consider your Enterprise Vault server itself. Because
changes can and do get made (new archiving policy, new retention rules, new
Roles-Based Administration controls), it is a Best Practice that the Enterprise
Vault server itself receive a regular backup.

7
Symantec Technology Network
7
 Enterprise Vault Backups

Content on your Enterprise Vault server that should be backed up is:

¾ System Registry
¾ Directory where Enterprise Vault is installed
¾ The entire registry for Administrator including local directories of:
• Local_System
• Current _User
¾ Any directory where you store any:
• Custom Forms
• EVPM scripts
• Modified IIS configuration files

While the Enterprise Vault server does not require nightly backups, given changes
do happen in production, it is a Best Practice to perform a backup at least once-a-
week on the above content.

Enterprise Vault Indexes (3)

Contained inside of your Enterprise Vault deployment is one to many AltaVista™

indexes. If you are unsure as to where your indexes are located, you can determine
the location in the properties of your Indexing Service in the VAC (Vault Admin
Console) Console. These indexes have one of the most important parts of your
archived content and must be backed up as a Best Practice on a very regular
basis. Scheduling of the backups is (covered later in this paper) very important to
insure the system is at a quiescent state for optimal integrity of the indexes whist
performing your backup. It is a mandatory Best Practice that you only backup the
indexes when Enterprise Vault services are either stopped, or running in “Read
Only” mode to insure an accurate backup of them.

Enterprise Vault Shopping Service Locations (3)

When users want to restore items manually contained in Enterprise Vault, their
content goes back to a Shopping Service location. This location is recommended to
receive a regular backup, but it is not necessarily required to back it up on the
same nightly schedule. Whilst the Shopping Service is quiescent during your backup
schedule, users will still be able to search content, but not restore it so it is a Best
Practice that the scheduling of your backups occurs during user off-hours. Unlike
other parts of your Enterprise Vault environment, it is not mandatory to backup
your Shopping Service locations, but if you do perform backups of them it is a Best
Practice that the Enterprise Vault servers are either stopped, or running in “Read
Only” mode.

Enterprise Vault Store Partitions (3)

Your archived content stored from its original location into Enterprise Vault will
typically be located as many flat files on an NTFS partition (exception: EMC
8
Symantec Technology Network
8
 Enterprise Vault Backups

Centera) with a folder directory structure cascading down. If you are unsure as to
where your Vault Store Partitions are located, you can determine the location in the
properties of them in the VAC (VAULT ADMIN CONSOLE) Console. As the content in
your Vault Store partitions is crucial to your environment, it is a Best Practice that
all archived content receives regular backups as disks and disk subsystem solutions
can fail. It is a mandatory Best Practice that you only backup the Vault Store
partitions when Enterprise Vault services are either stopped or running in “Read
Only” mode to insure an accurate backup of them.

If the size of your Vault Store Partitions is very large, you may wish to consider
options such as Symantec VVR, disk-replication (EMC Centera), SnapBackup
(Network Appliance Filers) or RoboCopy to aid in reducing end-user downtime /
access to archived content. Whilst the use of these technologies is outside the
technical scope of this paper, they are all very good solutions that should be
considered for larger sites as part of a Best Practice scenario for DR protection.

SQL Databases (2)

Contained in your Microsoft SQL environment are many databases that Enterprise
Vault uses as part of the solution. These databases are essential to insure that your
deployment is functional, able to archive and know where content is when you, or
your users access archived content. Because of the number of users, some sites will
have a dedicated SQL server for their Enterprise Vault environment; in other
smaller sites, shared access to a SQL server. Regardless of SQL installation type,
Best Practice dictates that you must make backups of the Enterprise Vault data
contained in SQL.

Of particular importance, the following Enterprise Vault databases as part of the

Best Practices must be backed up on a regular basis:

¾ Directory Database
¾ Audit Database
¾ Monitoring Database
¾ Database transaction logs
¾ Compliance &/or Discovery Accelerator Database(s)

For upgrades / Sp’s being applied to your SQL server, it is a Best Practice to make
sure that when a full backup is made that the master and msdb databases are
backed up as they contain information pointing to the existence of Enterprise Vault
on the SQL server.

In many sites, the above parts of your Enterprise Vault infrastructure may be
backed up by a separate person, such as a DBA (Database Administrator) which
can be sufficient but the scheduling (covered later in this paper) is critical for these
components to be properly secured.

When backing up SQL server(s), it is a Best Practice to:

9
Symantec Technology Network
9
 Enterprise Vault Backups

¾ Backup the Enterprise Vault databases at least once a week

¾ Truncate the transaction logs at the same time (to maintain
reasonable disk space consumption for the overall SQL server health.)
¾ Backup the database transaction log daily
¾ Backup the Enterprise Vault databases before / after any changes to
your Enterprise Vault environment (Service Pack, new release
installation, new target(s) being added)

Depending upon your Enterprise Vault architecture, you may have just one set of
databases, or in the case of our Compliance Accelerator / Discovery Accelerator
customers you may have additional databases. Both require that Best Practice
controls as part of your DR solution that they are be backed up each day as well.

Enterprise Vault Backup Timing and Scheduling

The previous diagrams went into detail at the network level for the various
components used within a simple Enterprise Vault environment. Timing is a crucial
part of a Best Practice for backing up your archived content to insure optimal
safety / DR preparedness for your environment.

Many resources are contended with by the “backup window” in an IT organisation

on a nightly basis, and each of these plays a role in determining how, when, and
what to backup.

Most customers are aware overall of activities on their infrastructure, but can fail to
take into account other activities that impair their ability to regularly successfully
get a good backup of their Enterprise Vault data. This can have side-effects during
a DR scenario that no one would want to encounter.

Important factors that can affect your backup timing include:

¾ Archiving Window
o The nightly window when Enterprise Vault is converting content
o When are users offline and online
¾ Network traffic
o Is the LAN or WAN capable of backing up large data sets
o What else is running over the LAN or WAN to the target server(s) to be
backed up?
¾ How much time do you have in a backup window
o Will any servers with Enterprise Vault data be unavailable
o Contention with other network activities
¾ When does Exchange &/or Domino
o Have its’ backups performed
o Compress mail stores or databases
¾ Will the backup be performed to
10
Symantec Technology Network
10
 Enterprise Vault Backups

o Tape
o NTFS disk
o Centera replication

Each of the above factors can and do have a direct effect on your ability to perform
your Enterprise Vault backups. Failing to counter in these type of network and
system activities is against Best Practice and should be avoided wherever
possible.

Overview of an evening for backups

In implementing your Best Practices for backups, we recommend you make a
chart that details significant activities as discussed in the previous section. A
suggested chart to give you good visibility is below.

Start End
Activity Time Time Best Practice Recommended Frequency
Archiving Daily
Window
MSMQ Daily
Buffer Drain
Window
Exchange Daily or Weekly – site specific
Backups
EV into
Read Only At Start of each time backups are performed
Mode
EV
Archive As dictated by local site rules, at least once a week
Backup
EV into
Normal At End of each time backups are performed
Mode
EV Pending Can run from end of Backups to 0900 / user arrival
Shortcut time in mornings or as part of archiving task
Conversion scheduling – site specific
EV SQL
Database
and Daily – Transaction Logs, Weekly EV Databases
Transaction
Log
Backups
EV Server As dictated by local site rules, at least
Backup once a month

The above chart is very handy to assist you in making sure that you have proper
justification, and inform all on your team of the activities to hand. The
recommendations in the chart vary slightly if you have Enterprise Vault safety
copies enabled, and would allow you to avoid the Pending Shortcut conversion for
sites that have chosen that option in the VAC (VAULT ADMIN CONSOLE).
11
Symantec Technology Network
11
 Enterprise Vault Backups

Addendum / Additional Information

It is important to know that several additional items in your Enterprise Vault
environment can have an effect on your backup windows, and should be reviewed
periodically as part of Best Practice to insure that you meet your SLA to your
business.

¾ Tape & Partition Management for LTO3 users

• If you setup your maximum partition size to be slightly less than the
current size of one tape (400GB) it will make administration and
management better for your time windows.
¾ Backup closed partitions
• Open partitions should still be backed up daily or at most weekly.
• Once a month if you have collections enabled as when an item in a
collection is no longer retained, the collections can decrease in size.
¾ Refresh tapes
• In sites where you do not do disk-based backups, we recommend you
cycle new tapes into your Enterprise Vault backup schedule as dictated by
local site rules.
• Insure proper tape destruction procedures are in effect to protect
IPR (Intellectual Property) contained on these tapes.
¾ Scheduling the start and stop of archiving
• Allow your Enterprise Vault server and MSMQ queues sufficient time to
finish any current activities so that you have an optimal backup image
taken.
• Factor in how much you can do in parallel.
¾ Use a good batch script to run backups
• Insure that all Enterprise Vault tasks must are setup to run on the
Enterprise Vault server as type “Automatic”
• We recommend as a Best Practice the script below from our Enterprise
Vault documentation at a minimum
• Allow a minimum time of 10 – 15 minutes before / after each script
before starting backups.

The following scripts are provided as an example to guide in performing Best

Practices to prepare your Enterprise Vault environment for backing up the data
contained in it. The script “PREBACKUP.BAT” stops your Enterprise Vault services
from archiving to be ready to backup content. The script “POSTBACKUP.BAT”
changes Enterprise Vault back into normal operational mode once your backups are
done.

12
Symantec Technology Network
12
 Enterprise Vault Backups

PREBACKUP.BAT:
REM ---------------------------------
REM prebackup.bat
REM This script is to put EV into read-only mode so we can run backups
REM ---------------------------------
net stop /y "Enterprise Vault Task Controller Service"
net stop /y "Enterprise Vault Storage Service"
net stop /y "Enterprise Vault Indexing Service"
net stop /y "Enterprise Vault Shopping Service"
regedit /s c:\readonly.reg
net start "Enterprise Vault Storage Service"
net start "Enterprise Vault Indexing Service"
net start "Enterprise Vault Shopping Service"
net start "Enterprise Vault Task Controller Service"

POSTBACKUP.BAT:
REM ---------------------------------
REM postbackup.bat
REM This script is to put EV into read-write mode after running backups
REM ---------------------------------
net stop /y "Enterprise Vault Storage Service"
net stop /y "Enterprise Vault Indexing Service"
net stop /y "Enterprise Vault Task Controller Service"
regedit /s c:\normal.reg
net start "Enterprise Vault Storage Service"
net start "Enterprise Vault Indexing Service"
net start "Enterprise Vault Task Controller Service"

Customers using Enterprise Vault FSA (File System Archiving) need to run an
additional script when a 3rd party backup application accesses a file-system with
Placeholders on it so the application will respect the “Archive Bit” flag that FSA
uses.

FSABACKUP.BAT
REM ---------------------------------
REM fsabackup.bat
REM This script is to put FSA into non-recall mode to allow applications to
REM backup a file system containing PlaceHolders
REM ---------------------------------
EVFSABackupmode.exe backup (or normal) <Server> [DirectoryComputer]
REM

The above script must be run in “backup” mode to stop the Placeholder service, and
“normal” mode to start the Placeholder service on each target FSA server.

13
Symantec Technology Network
13
 Enterprise Vault Backups

14
Symantec Technology Network
14
 Enterprise Vault Backups

About the Author

Steve Blair was the founder of Product Management in KVS before acquisition by VERITAS in
late 2004. His role has been pivotal in helping the Enterprise Vault team expand and grow the
products over the past two years. Steve’s responsibilities include Product Strategy, Core
functionality, Storage, Internationalization, and Product Evangelist. Presently Steve is the Sr.
Regional Product Manager for Enterprise Vault in Asia Pacific region and is based in Sydney
Australia. Prior to joining KVS, Steve was involved in technology assessment and advisement
for a variety of UK and EMEA based Venture Capital firms advising investors and boards on
product strategy, and how to best grow emerging technology firms from concept to reality. In
prior positions, Steve was a Global Director in IT for a multi-national German bank for
Enterprise System Management for the investment division of the bank. He occupied the role of
Chief of Technology for a MoD certified Managed Services provider across EMEA. In other
startups, Steve spent many years from pre to post IPO one of the Principal Engineers doing R&D
for new products in areas of distributed systems theory, as well as network security and was lead
or co-inventor for many patented inventions to aid IT groups in more efficiently doing their jobs.
He has over 22 years of experience with IT and development activities. He has worked in
Hardware and Software development, Systems Administration, Network Operations, and
Enterprise Systems management.

About Symantec Technology Network (STN)

Symantec Technology Network (STN) is Symantec Corporation’s technical information

generation and dissemination organization. It distributes a free monthly technical newsletter that
discusses timely technology events to 120,000 email subscribers across the globe. STN also
publishes technical data storage and security articles each month for large enterprise and Small
and Medium Business (SMB) readers, as well as hosts a variety of blogs and product discussion
forums discussing Symantec product tips and insights. To subscribe to STN’s Technical
Newsletter and review other STN materials, please visit STN at:

http://www.symantec.com/enterprise/stn/index.jsp

W. David Schwaderer is STN’s Editor-In-Chief. He has authored ten technical books, and six
commercial software programs. His eleventh book titled Innovation Survival is nearing
completion. He has a Masters Degree in Applied Mathematics from the California Institute of
Technology and an MBA from the University of Southern California. He lectures at Stanford
and in Silicon Valley on the subject of innovation.

Russ G. Gregg is STN’s Video Content Producer. He is presently focusing on producing state-
of-the-art digital video for the Web and small form-factor, portable video devices such as Apple
Inc.’s iPhone™ and Video iPod®. Russ has a Bachelor of Arts in English Literature, Texas from
A&M University.
15
Symantec Technology Network
15

About Symantec
Symantec is the world leader
in providing solutions to help
individuals and enterprises
assure the security, availability, and
integrity of their information.
Headquartered in Cupertino,
Calif., Symantec has operations
in more than 40 countries.
More information is available at
www.symantec.com.
Symantec Technology Network
Symantec Corporation
World Headquarters
20330 Stevens Creek Boulevard
Cupertino, CA 95014 USA
408 517 8000
800 721 3934
16
Enterprise Vault Backups
Symantec and the Symantec logo are U.S. registered trademarks
of Symantec Corporation. Other brands and products are
trademarks of their respective holder/s. Any technical information
that is made available by Symantec Corporation is the copyrighted
work of Symantec Corporation and is owned by Symantec
Corporation. NO WARRANTY. The technical information is
being delivered to you as-is and Symantec Corporation makes no
warranty as to its accuracy or use. Any use of the technical
documentation or the information contained herein is at the risk of
the user Copyright © 2007 Symantec Corporation All rights
16