Sunteți pe pagina 1din 9
Keys to optimizing your backup environment: Veritas NetBackup Jeff Harbert Storage Consultant GlassHouse
Keys to optimizing your
backup environment:
Veritas NetBackup
Jeff Harbert
Storage Consultant
GlassHouse Technologies, Inc.
jharbert@glasshouse.com
Introduction Audience Profile Storage Management Interdependence Case Study Backup Architecture and Performance
Introduction
Audience Profile
Storage Management Interdependence
Case Study
Backup Architecture and Performance
• Server
• Tape
• Network
• Clients
Audience Response Are you a NetBackup Administrator? • Yes • No How long have you
Audience Response
Are you a NetBackup Administrator?
• Yes
• No
How long have you been using the product?
• 1. 0-6 months
• 2. 6-12 months
• 3. 12-24 months
• 4. More than 2 years
Case Study – Biotechnology Company Technical Issues Business Problems Backup architecture unable to scale with
Case Study – Biotechnology Company
Technical Issues
Business Problems
Backup architecture unable to
scale with increasing storage
capacities and server growth
Limited budget and staff
Network architecture unable to
support throughputs required
for backup
Reliability and performance
problems for both hardware
and software
Regulatory / data retention
requirements
Lack of NBU / OS tuning:
Implementation did not include
optimization
Disconnect between application
design and storage
management
Client-side issues impacting
backup success (OS, antivirus,
network, applications)
Shrinking backup windows
Lack of data management
policies
NetBackup Architecture Tape/SAN/Disk Tape/SAN/Disk S D SD HEWLETT HEWLETT PACKARD PACKARD Technology Technology
NetBackup Architecture
Tape/SAN/Disk Tape/SAN/Disk
S D
SD
HEWLETT
HEWLETT
PACKARD
PACKARD
Technology Technology
NetBackup
NetBackup
Server
Server
Clients Clients
SD
SD
5411 5411
Enterprise Switch
Enterprise Switch
10 10
00 BASE X
00 BASE X
1 1
2
2
AMBER =
AMBER =
AC TIVI TY
AC TIVI TY
GREEN
GREEN
=
=
LINK
LINK
O K
O K
FL ASHING GREEN =
FL ASHING GREEN =
DISABLED DISABLED
10/ 100 BASE TX
10/ 100 BASE TX
4 4
5 5
6
6
7
7
8
8
9
9
1 0 11
1 0 11
1
1
2 2
3 3
3 3
POWER
POWER
1 1
2 2
3 3
3 3
MGMT
MGMT
4 4
5
5
6
6
7
7
8
8
9 9
10
10
11
11
R R
3R
3R
3 3
Network Network
Media Media
Server Server
Case Study Technical Issues Solution Backup server underpowered for size of environment Upgraded hardware to
Case Study
Technical Issues
Solution
Backup server underpowered
for size of environment
Upgraded hardware to HP
rp7410
Lacking in CPUs, RAM, and
Network throughput
Master: 4GBs of RAM, 4
CPUs, Gb Ethernet cards
No media servers to balance
load
Partitioned server into master
and media server
Upgraded hosts with more than
200 GBs into SAN media
servers
NetBackup not properly tuned
for optimal performance
Modified NetBackup server-side
variables to maximize
performance
NetBackup Server Performance Application Performance Tuning • Backup Schedule Balancing • Number of Backup Jobs
NetBackup Server Performance
Application Performance Tuning
• Backup Schedule Balancing
• Number of Backup Jobs
Multi-streaming and multiplexing
• Specific Tuning Variables
NET_BUFFER_SZ
DISABLE_RESOURCE_BUSY
CLEAN_IN_BACKGROUND
Pre-Process Interval
NetBackup Performance Tuning Guide
• UNIX: http://seer.support.veritas.com/docs/240733.htm
• Windows:
http://seer.support.veritas.com/docs/248373.htm
Average Media Server Volume 1,200 1,000 800 Server 1 600 Server 2 Server 3 400
Average Media Server Volume
1,200
1,000
800
Server 1
600
Server 2
Server 3
400
Server 4
200
Server 4
0
Server 3
Server 2
Server 1
Data Transferred (GB)
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
NetBackup Architecture Tape/SAN/Disk Tape/SAN/Disk S D SD HEWLETT HEWLETT PACKARD PACKARD Technology Technology
NetBackup Architecture
Tape/SAN/Disk Tape/SAN/Disk
S D
SD
HEWLETT
HEWLETT
PACKARD
PACKARD
Technology Technology
NetBackup NetBackup
Server Server
Clients Clients
SD
SD
5411 5411
Enterprise Switch
Enterprise Switch
10 10
00 BASE X
00 BASE X
1 1
2
2
AMBER =
AMBER =
AC TIVI TY
AC TIVI TY
GREEN
GREEN
=
=
LINK
LINK
O K
O K
FL ASHING GREEN DISABLED DISABLED =
FL ASHING GREEN =
10/ 100 BASE TX
10/ 100 BASE TX
4 4
5 5
6
6
7
7
8
8
9
9
1 0 11
1 0 11
1
1
2 2
3 3
3 3
POWER
POWER
1 1
2 2
3 3
3 3
MGMT
MGMT
4 4
5
5
6
6
7
7
8
8
9 9
10
10
11
11
R R
3R
3R
3 3
Network Network
Media
Media
Server
Server
Case Study Technical Issues Solution Veritas Shared Storage Option deployed with STK L700 and 20
Case Study
Technical Issues
Solution
Veritas Shared Storage Option
deployed with STK L700 and
20 LTO-1 drives
Modified SSO to limit drive
sharing to smaller number of
servers
Frequent drive problems
caused multiple backup
failures
Updated persistent bindings to
minimize impact of server
reboots and SCSI-locks
Overall Throughput was less
than expected
Tape drives were “shoe-
shining” due to a lack of data
Increased the number of multiple
streams written to each drive
Reduced the number of drives
available for backup
Modified SIZE_DATA_BUFFERS and
NUMBER_DATA_BUFFERS variables
on media servers
Tape Tuning Strategies Storage Unit Fragment Size • Rule-of-Thumb = 2GB Multiplexing Data • Reduces
Tape Tuning Strategies
Storage Unit Fragment Size
• Rule-of-Thumb = 2GB
Multiplexing Data
• Reduces “shoe-shining” effect
• Too much can negatively impact restore speed.
In-Line Duplication
• Requires additional hardware, but enables tapes to be
sent offsite quicker
Software vs. Hardware Compression
• Best Practice: Utilize hardware compression unless
network-constrained.
Disk-Based Backups with NetBackup v5.0 Enhanced functionality of v5.0 makes disk-based backups a viable option.
Disk-Based Backups with NetBackup v5.0
Enhanced functionality of v5.0 makes disk-based
backups a viable option.
Disk as a Primary Storage Device
• Enterprise-class solutions are limited due to tremendous
storage requirements.
Disk-Based Backups with NetBackup v5.0 (II) Disk as a Staging Device • v5.0 includes Staging
Disk-Based Backups with NetBackup v5.0
(II)
Disk as a Staging Device
• v5.0 includes Staging functionality.
• Data is automatically moved based on pre-
established policies.
• Leverages speed of disk for backup, with tape for
long-term storage
Disk as Hybrid-Solution
• Direct some backups to disk, others to tape.
• Eliminates additional step of staging, but provides
advantages of both disk and tape
Storage Unit Distribution Amount of Data Transferred for a 7-Day Period (15.4 TB) STK9940A531, 0,
Storage Unit Distribution
Amount of Data Transferred for a 7-Day Period (15.4 TB)
STK9940A531, 0, 0%
STK9940A317, 0.3TB, 2%
STK9940A258, 0.4TB, 2%
STK9940B317, 1.1TB, 7%
STK9940B258, 5.9TB, 39%
STK9940B531, 3.6TB, 23%
STK9940B532, 4TB, 27%
NetBackup Architecture Tape/SAN/Disk Tape/SAN/Disk S D SD HEWLETT HEWLETT PACKARD PACKARD Technology Technology
NetBackup Architecture
Tape/SAN/Disk Tape/SAN/Disk
S D
SD
HEWLETT
HEWLETT
PACKARD
PACKARD
Technology Technology
NetBackup NetBackup
Server Server
Clients Clients
SD
SD
5411 5411
Enterprise Switch
Enterprise Switch
10 10
00 BASE X
00 BASE X
1 1
2
2
AMBER =
AMBER =
AC TIVI TY
AC TIVI TY
GREEN
GREEN
=
=
LINK
LINK
O K
O K
FL ASHING GREEN =
FL ASHING GREEN =
DISABLED DISABLED
10/ 100 BASE TX
10/ 100 BASE TX
4 4
5 5
6
6
7
7
8
8
9
9
1 0 11
1 0 11
1
1
2 2
3 3
3 3
POWER
POWER
1 1
2 2
3 3
3 3
MGMT
MGMT
4 4
5
5
6
6
7
7
8
8
9 9
10
10
11
11
R R
3R
3R
3 3
Network Network
Media Media
Server Server
Case Study Technical Issues Solution Separate Backup Network was created to reduce impact on corporate
Case Study
Technical Issues
Solution
Separate Backup Network was
created to reduce impact on
corporate network
Modified “Required Interface”
option to bind client backups to
appropriate network interface
Clients did not always use backup
network for data transfer
Intermittent backup failures
following host reboots
Hard-code network settings to
100 Mb, Full-Duplex
NICs and Ports set to “auto-
negotiate”
Log files indicate that Media
servers were “waiting for full
buffers” during network
backups
Tuned NET_BUFFER_SZ
Variable on both media servers
and clients
Slowed backup performance
Network Design Dedicated Private VLAN architectures • Logical network segregation & physical network segregation
Network Design
Dedicated Private VLAN architectures
• Logical network segregation & physical network
segregation
• Use jumbo frames and gigabit VLANs.
Improves streaming efficiencies to tape
Multiple NICs for client network load balancing
• Etherchannel configurations
• Load balancing for network, switches, server HBAs
Network Design, II NetBackup Tuning Variable • NET_BUFFER_SZ: Determines size of network communications buffer used
Network Design, II
NetBackup Tuning Variable
• NET_BUFFER_SZ: Determines size of network
communications buffer used to receive data from
network
If set too small, media servers can’t empty
buffers fast enough.
If set too large, media servers have to wait for
data.
Client settings should match media servers.
Sample Charts Available From GlassHouse Buffer Information Summary Data Buffer Analysis per Media Server 50,000
Sample Charts Available From GlassHouse
Buffer Information Summary
Data Buffer Analysis per Media Server
50,000
45,000
Waited for full buffer
Waited for empty buffer
40,000
35,000
30,000
25,000
20,000
15,000
10,000
5,000
0
Waited for empty buffer
Waited for full buffer
Media Servers
Number of Times
Server 1
Server 2
Server 3
Server 4
NetBackup Architecture Tape/SAN/Disk Tape/SAN/Disk S D SD HEWLETT HEWLETT PACKARD PACKARD Technology Technology
NetBackup Architecture
Tape/SAN/Disk Tape/SAN/Disk
S D
SD
HEWLETT
HEWLETT
PACKARD
PACKARD
Technology Technology
NetBackup NetBackup
Server Server
Clients Clients
SD
SD
5411 5411
Enterprise Switch
Enterprise Switch
10 10
00 BASE X
00 BASE X
1 1
2
2
AMBER =
AMBER =
AC TIVI TY
AC TIVI TY
GREEN
GREEN
=
=
LINK
LINK
O K
O K
FL ASHING GREEN =
FL ASHING GREEN DISABLED DISABLED =
10/ 100 BASE TX
10/ 100 BASE TX
4 4
5 5
6
6
7
7
8
8
9
9
1 0 11
1 0 11
1
1
2 2
3 3
3 3
POWER
POWER
1 1
2 2
3 3
3 3
MGMT
MGMT
4 4
5
5
6
6
7
7
8
8
9 9
10
10
11
11
R R
3R
3R
3 3
Network Network
Media Media
Server Server
Case Study Technical Issues Solution Oracle Agent was implemented for Hot Oracle Database Backups Implemented
Case Study
Technical Issues
Solution
Oracle Agent was implemented
for Hot Oracle Database
Backups
Implemented Block-Level
Incremental backups of Oracle
30% Faster than standard RMAN
backups
Backups of some servers ran out-
of-window
Network Appliance Filers
backed up via NFS mounts
Implemented NDMP backups to
dedicated tape drives
Full backups took 5 days
Testing indicated that 3 rd -Party
Remote backups were significantly
slower
Backups of Netware clients ran
out-of-window
Tuned Netware variables per
Veritas-recommended settings
Reduced backup times by 33%
Client Considerations Include/Exclude list size • Maintenance is difficult, but necessary. • Exclude Open Files,
Client Considerations
Include/Exclude list size
• Maintenance is difficult, but necessary.
• Exclude Open Files, Temp files, etc.
Open File Processing
Block-Level / Flash Backups
Client options settings
• Pre-Process Interval (set on master server) -
Increasing default interval
Number and size of files
Frequency of data change
Client Backup Performance Backup Jobs over 1 GB in Volume 16.0 14.0 12.0 10.0 8.0
Client Backup Performance
Backup Jobs over 1 GB in Volume
16.0
14.0
12.0
10.0
8.0
6.0
4.0
2.0
0.0
Throughput (MB/S)
Client 1
Client 2
Client 3
Client 4
Client 5
Client 6
Client 7
Client 8
Client 9
Client 10
Client 11
Client 12
Client 13
Client 14
Client 15
Client 16
Client 17
Client 18
To receive charts similar to those seen in this presentation specific to your environment go
To receive charts similar to those seen in
this presentation specific to your
environment go to:
www.glasshouse.com/backup
For a “cheat sheet” see the
highlighter in your conference bag.
Questions? Thank you. See me at Ask the Expert – Tuesday 5:00-6:00 PM
Questions?
Thank you.
See me at Ask the Expert –
Tuesday 5:00-6:00 PM