Sunteți pe pagina 1din 12
Managing Vertica clusters in a SaaS environment, Part 1: Copying Vertica databases across the cluster
Managing Vertica clusters in a SaaS environment, Part 1: Copying Vertica databases across the cluster

Managing Vertica clusters in a SaaS environment, Part 1: Copying Vertica databases across the cluster

Amol D. Barsagade (amolbarsagade@in.ibm.com) Software Architect IBM

03 June 2015

In the SaaS environment, data information and services are the key to customer success. HP® Vertica® is a critical database technology that assists customers to better manage their data in a cluster environment. This article illustrates the step-by-step approach for Vertica installation, configuration, and setup across the clusters. As a part of the setup, it is important to complete the installation and configuration and then to copy the database across the

Introduction

NIOD is an IBM® Unica® product under Enterprise Marketing Management (EMM) which serves customers in a SaaS environment that uses Hewlett-Packard® Vertica clusters. The first step in setting up the Vertica database is to install Vertica on the target cluster. Each source and target cluster has three nodes. The required packages need to be verified, and the prerequisites for copying the database across the cluster need to be followed. An empty database in the target cluster is created. The database then is copied from the source to the target cluster.

Logging in to the source and target Vertica cluster nodes

Ensure that the passwordless SSH for the

source and target cluster nodes. The

target nodes without a password. Also, this user must be able to log in in the reverse order, that is, from all target cluster nodes to all the source cluster nodes.

dbadmin dbadmin
dbadmin
dbadmin

user works between and across each of the

user must be able to log in to all the source and

For simplicity, assume that the source and target Vertica clusters and nodes are as follows:

Table 1. Source and target Vertica cluster details

Cluster Type

Cluster Node Name

IP

DB_Name

User

Source

source01.niod.unica.net

10.10.10.1 unicl01

 

dbadmin

 

source02.niod.unica.net

10.10.10.2 unicl01

 

dbadmin

© Copyright IBM Corporation 2015 Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Trademarks Page 1 of 12

developerWorks®

ibm.com/developerWorks/

 

source03.niod.unica.net

10.10.10.3

unicl01

dbadmin

Target

target01.niod.unica.net

10.10.10.4

 

dbadmin

 

target02.niod.unica.net

10.10.10.5

 

dbadmin

 

target03.niod.unica.net

10.10.10.6

 

dbadmin

Copy the Vertica installer to the target cluster node

Assume that the Vertica binary file

cluster and can be copied to the target cluster.

vertica-5.0.12-0.x86_64.RHEL5.rpm

is available on the source

Copy the binary file from the source cluster to the first node of the target cluster,

target01.niod.unica.net (10.10.10.4)

.

For example,:

[ambarsag@source01 Vertica-binaries]$ scp vertica-5.0.12-0.x86_64.RHEL5.rpm ambarsag@10.10.10.4:/local/ Vertica-binaries

Note: Verify that the binary file was copied to the target cluster node and give full permissions so

that the

dbadmin
dbadmin

user can run it.

[ambarsag@target01 Vertica-binaries]$ chmod -R 777 vertica-5.0.12-0.x86_64.RHEL5.rpm

Verify packages on the target cluster

Verify that the required packages and files exist on each node of the target cluster.

Verification of sudo and Python

Verify that sudo and Python are available on all the target cluster hosts. Verify that you have privileges to use them on these hosts by running the following command at a shell prompt:

[root@target01 dbadmin]# which sudo /usr/bin/sudo [root@target01 /]# which python /usr/bin/python

[root@target02 ~]# which sudo /usr/bin/sudo [root@target02 /]# which python /usr/bin/python

[root@target03 ~]# which sudo /usr/bin/sudo [root@target03 /]# which python /usr/bin/python

Verification of rsync and version

Verify that rsync 3.0.5 or later is installed on all cluster hosts.

[root@target01 /]# which rsync /usr/bin/rsync [root@target01 /]# rsync --version rsync version 3.0.6 protocol version 30

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 2 of 12

ibm.com/developerWorks/

developerWorks®

Note: Verify the same on the other two nodes,

target02

and

target03

.

Verification of the su file

 

Verify that the

su
su

file exists in

/etc/pam.d

on all the cluster nodes.

 

[root@target01 pam.d]# pwd /etc/pam.d

 

Note: Verify the same on the other two nodes,

target02

and

target03

.

Verification of NTP demon

Verify that the NTP Demon is running on all the target cluster nodes.

Note: If the NTP demon is installed, the output resembles:

The output indicates the run levels where the demon runs. Verify that the current run
The output indicates the run levels where the demon runs. Verify that the current run level of the system
(usually 3 or 5) has the NTP demon set to ON.
[root@target01 pam.d]# chkconfig --list ntpd
ntpd
0:off
1:off
2:on
3:on
4:on
5:on
6:off
[root@target01 /]# runlevel
N 3

Note: Verify the same on the other two nodes,

target02 and target03 .

Define the loopback address

Ensure that the

with only the name

/etc/hosts localhost
/etc/hosts
localhost

file exists. Ensure that the file contains the loopback address 127.0.0.1

in both fully qualified and unqualified forms.

For example:

127.0.0.1 localhost.localdomain localhost

Note: Verify the same on the other two nodes,

Setting up the host file

Setup the /etc/hosts file on each node of the cluster

Make sure that the

target02 and target03 .

/etc/hosts

file includes all of the hosts that become part of the cluster.

[root@target01 /]# vi /etc/hosts

Input the following entry into the hosts file:

#Vertica HB

10.10.10.4

target01.niod.unica.net

10.10.10.5

target02.niod.unica.net

10.10.10.6

target03.niod.unica.net

Note: Verify the same on the other two nodes,

target02 and target03 .

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 3 of 12

developerWorks®

Ensure port 5433 is not in use

ibm.com/developerWorks/

[root@target01 /]# netstat -atup [root@target01 /]# vi /etc/services

This command does not list the port number if it is identified by a service
This command does not list the port number if it is identified by a service that is defined in
/
etc/services
. Some common services that might use port 5433 include
pyrrho
,
nibbler
, and
postgresql
.

Note: Verify the same on the other two nodes,

Ensure /dev/pts is mounted

target02 and target03 .

[root@target01 /]# cd /dev/pts [root@target02 /]# cd /dev/pts [root@target03 /]# cd /dev/pts

Installing Vertica on the target cluster

Choose one of the cluster hosts to be the Administration Host for the purposes of installing Vertica

and for using the Vertica Administration Tools. Here, the node for the installation.

target01.niod.unica.net

is used

Run the RPM package installer

Note: Since

package installer.

sudo
sudo

is used to log in to the root,

sudo su - , sudo
sudo su -
, sudo

must also be used to run the RPM

[root@target01 Vertica-binaries]# sudo rpm -Uvh /local/Vertica-binaries/vertica-5.0.12-0.x86_64.RHEL5.rpm

Preparing

1:vertica

########################################### [100%] ########################################### [100%]

Vertica Analytic Database V5.0.12-0 successfully installed on host target01.niod.unica.net

Note: If you are root, then use the following command:

# rpm -Uvh pathname

Note: This command creates a folder

/opt/vertica .

[root@target01 vertica]# pwd /opt/vertica [root@target01 vertica]# ls bin config examples include java lib lib64 log oss packages sbin scripts sdk share spread

Run the install script

To start the installation, run the install script from the

/opt/vertica/sbin

location.

Note: If you run the Vertica installation script without parameters, the script performs a single-node

installation on

localhost

.

If you are root:

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 4 of 12

ibm.com/developerWorks/

developerWorks®

# /opt/vertica/sbin/install_vertica -s host_list -r rpm_package -u dba_username

If you are using sudo:

$ sudo /opt/vertica/sbin/install_vertica -s host_list -r rpm_package -u dba_username

Note: The

-r
-r

and

-s
-s

parameters are not required for single-node installations.

Example:

[ambarsag@target01 sbin]$ sudo /opt/vertica/sbin/install_vertica -s 10.10.10.4,10.10.10.5,10.10.10.6 -r /local/Vertica-binaries/ vertica-5.0.12-0.x86_64.RHEL5.rpm -u dbadmin -p xxxx -g dbadmin

where:

• -s : host_list • -r : rpm_package • -u : dbadmin user • -p
• -s
: host_list
• -r
: rpm_package
• -u
: dbadmin
user
• -p
: password for
dbadmin
• -g
: group of
dbadmin
user

user

On successful completion of the installation you get the following message:

Verifying spread configuration on whole cluster Creating node node0001 definition for host 10.10.10.4 Done Creating node node0002 definition for host 10.10.10.5 Done Creating node node0003 definition for host 10.10.10.6 Done Error Monitor 0 errors 0 warnings Installation complete.

Prerequisites for copying the database from the source to the target cluster

To copy the source database followed.

unicl01
unicl01

to the target cluster nodes, the following guidelines must be

Have the same number of nodes as the source cluster

Make sure that the source and the target Vertica clusters have same number of nodes.

Be accessible from the source cluster

You might need to add the host name of each target cluster node into the hosts file of the source cluster. Since this copy of the database is initiated from one of the source cluster nodes, all the source node host files must contain the target host names.

On the first node of the source cluster,

source01.niod.unica.net

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

, log in as root.

Page 5 of 12

developerWorks®

ibm.com/developerWorks/

[root@source01 /]# vi /etc/hosts

#Vertica SOURCE

10.10.10.1

source01.niod.unica.net

10.10.10.2

source02.niod.unica.net

10.10.10.3

source03.niod.unica.net

#Vertica TARGET

10.10.10.4

target01.niod.unica.net

10.10.10.5

target02.niod.unica.net

10.10.10.6

target03.niod.unica.net

Note: Verify the same on the other two nodes,

andNote: Verify the same on the other two nodes, source03 . Copy the license file from

source03 .
source03
.

Copy the license file from the source cluster to the target

The license file,

license file is on the source at

copied to each target cluster node at the

license.key

, must be copied to all the target cluster nodes. Assume that the

/opt/vertica/config/share

. In this example, the license file is

/opt/vertica/config/share

location.

Example:

source01:/opt/vertica/config/share $ ls -la -rw-rw-rw- 1 dbadmin verticadba 547 Nov 16 2011 license.key

source01:/home/dbadmin $ scp license.key dbadmin@10.10.10.4:/opt/vertica/config/share

Note: Similarly, copy the license file to the other two nodes of the target cluster, 10.10.10.5 and

10.10.10.6.

Creating an empty database on the target cluster

Now, create an empty database on the target cluster with the same name,

cluster. Have a database with the same name as the database that is being copied. This database can be empty.

unicl01
unicl01

, as the source

Steps to create an empty database on the target cluster:

• Log in to

• Run Vertica Admintools as follows:

target01.niod.unica.net

as the

user.Admintools as follows: target01.niod.unica.net as the [dbadmin@target01 bin]$ /opt/vertica/bin/admintools • This

[dbadmin@target01 bin]$ /opt/vertica/bin/admintools

• This opens the "Main Menu" window.

• Select "Configuration Menu".

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 6 of 12

ibm.com/developerWorks/

Figure 1. Configuration Menu

developerWorks®

Figure 1. Configuration Menu developerWorks® • Click OK. • On the "Configuration Menu"

• Click OK.

• On the "Configuration Menu" window, click "Create Database".

Figure 2. Create Database

click "Create Database". Figure 2. Create Database • Click OK. • Enter the name of the

• Click OK.

• Enter the name of the database and an optional comment.

dbname: unicl01 comment: Copy db from source to target vertica cluster.

• Click OK.

• Enter a password for the new database.

• Click OK.

• Reenter the same password and click OK.

• Select the list of target cluster nodes on which you want to create the same database.

[X]

10.10.10.4

[X]

10.10.10.5

[X]

10.10.10.6

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 7 of 12

developerWorks®

ibm.com/developerWorks/

• Catalog path name:

• Data path name:

• Click OK.

• On the "Current Database Definition" window, review the details and click "Yes".

/home/dbadmin /home/dbadmin
/home/dbadmin
/home/dbadmin

Note: The database

database from the Vertica Admintools and can connect, start, and stop the database.

unicl01
unicl01

is created on all three nodes of the target cluster. You can view the

Copying the Vertica database from the source to the target cluster

Create the backup directories on all the three target cluster nodes. Then, create a configuration file

to be run along with the

copycluster command.

Create the Backup directory on target nodes

Create the backup directory on all the three nodes of the target cluster as

Then, change the owner to

dbadmin .
dbadmin
.

/home/dbadmin/backups

.

[root@target01 /]# mkdir /home/dbadmin/backups [root@target01 /]# chown -R dbadmin:dbadmin /home/dbadmin/backups

Note: Create similar directories on the other two nodes of this cluster,

target02 and target03 .

Identify the database node name.

Identify the database node names on the source cluster that are to be used in the configuration file

by running the copycluster

command. For example:

source01:/opt/vertica/config $ /opt/vertica/bin/admintools -t node_map -d unicl01

DATABASE

--------------------------------------------------------

| NODENAME

| HOSTNAME

unicl01 | v_unicl01_node0001

| 10.10.10.1

unicl01 | v_unicl01_node0002

| 10.10.10.2

unicl01 | v_unicl01_node0003

| 10.10.10.3

Note: Here, the name that follows the

-d
-d

flag is the database name.

Create a configuration file

You need to create a configuration file,

CopyCluster.ini

source01.niod.unica.net

. For example:

, on the source node,

File name: CopyCluster.ini Location: /opt/vertica/config/CopyCluster.ini

Sample Contents of

CopyCluster.ini

[Misc] snapshotName = CopySourceDB tempDir = /tmp verticaConfig = False retryCount = 5 retryDelay = 1

[Database]

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 8 of 12

ibm.com/developerWorks/

developerWorks®

dbName = unicl01 dbUser = dbadmin dbPassword = xxxx dbPromptForPassword = False [Transmission] encrypt = False checksum = False port_rsync = 50000 bwlimit = 0 hardLinkLocal = False

[Mapping0]

dbNode = v_unicl01_node0001 backupHost = 10.10.10.4 backupDir = /home/dbadmin/backups

[Mapping1]

dbNode = v_unicl01_node0002 backupHost = 10.10.10.5 backupDir = /home/dbadmin/backups

[Mapping2]

dbNode = v_unicl01_node0003 backupHost = 10.10.10.6 backupDir = /home/dbadmin/backups

Note:

Node_Name= /home/dbadmin/backups Note: • • • • • • of each source cluster. backupHost : The

of each source cluster.

backupHost : The target cluster node name. backupDir : Back up directory on target cluster node.

dbName : dbUser : dbPassword
dbName
:
dbUser
:
dbPassword

Name of the source database,

unicl01 .
unicl01
.

Administrator user, the same on the source and the target cluster.

: Password of source database.

Run the copycluster command

To copy the cluster, you need to run administrator account, passing it the

file specifically to copy the database, use the where it is located.

vbr.py

from a node in the source database. Use the database

command. If you created a configuration

--task copycluster

--config-file

parameter to tell the backup utility

Run the following command as

fromto tell the backup utility Run the following command as config/CopyCluster.ini . source01.niod.unica.net . Browse

config/CopyCluster.ini

.

source01.niod.unica.net

. Browse to

/opt/vertica/

source01:/home/dbadmin $ cd /opt/vertica/config source01:/opt/vertica/config $ vbr.py --config-file CopyCluster.ini --task copycluster

Note: Here you might get the following error:

Node v_unicl01_node0001 of target cluster is not DOWN; cannot copycluster. copycluster failed!

You need to stop the database

This action stops the database on the other two nodes as well. After you stop the database, run

onother two nodes as well. After you stop the database, run source01.niod.unica.net by using Vertica Admintools.

source01.niod.unica.net

by using Vertica Admintools.

the CopyCluster

command again.

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 9 of 12

developerWorks®

ibm.com/developerWorks/

source01:/opt/vertica/config $ vbr.py --config-file CopyCluster.ini --task copycluster

You might get the following message based on the size of the source database:

Copying 151093501882 out of 151093501882, 100%

Upon successful completion of copying the database from the source to the target cluster, you can start the database. Start the database from any one node of the target cluster nodes by using Vertica Admintools.

Conclusion

This article describes the importance of various Vertica packages that are required for the installation. It also helps to understand various configurations and the settings that are required for the installation. The step-by-step approach to copying the Vertica databases of IBM NIOD customers in a SaaS environment includes creating an empty database, setting the configuration

file, and running the

Acknowledgment

copycluster command by using vbr.py .

I would like to extend appreciation to Marjana Ivkovic (mivkovic@us.ibm.com), Team Lead, EMM SaaS Ops Applications Architects, for the continuous support and valuable technical guidance for completing this article.

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 10 of 12

ibm.com/developerWorks/

Resources

Learn

developerWorks®

• "Direct Integration with WebSphere Commerce" provides information on the direct integration model of OMS and WebSphere Commerce.

• "Sterling Definition Framework" for information on the Sterling SDF.

• "Agent and integration servers" for information on the Sterling agent servers and integration servers.

• In the Commerce Zone, get information about other Commerce resources.

• Stay current with developerWorks technical events focused on a variety of IBM products and IT industry topics.

• Watch developerWorks on-demand demos ranging from product installation and setup demos for beginners, to advanced functionality for experienced developers.

Get products and technologies

Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment.

Discuss

• Get involved in the My developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 11 of 12

developerWorks®

ibm.com/developerWorks/

About the author

Amol D. Barsagade

ibm.com/developerWorks/ About the author Amol D. Barsagade Amol D. Barsagade is a Database Architect with EMM

Amol D. Barsagade is a Database Architect with EMM SaaS Ops, IBM Experience One, India Software Lab, Pune. Amol provides database solutions for database architecture, performance tuning, capacity management, application development, and migrations to various partners and customers in Asia-Pacific. Amol has a bachelor's degree in computer science and several years of experience working with relational database systems, including DB2®, Oracle, and SQL server. Since 2006, he has made valuable contributions to IBM developerWorks by reviewing and publishing articles.

© Copyright IBM Corporation 2015 (www.ibm.com/legal/copytrade.shtml) Trademarks (www.ibm.com/developerworks/ibm/trademarks/)

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 12 of 12