Huawei SAP HANA Appliance Two Node Installation Guide (CH121&CH242&2288H&2488H&9008 V5) 08

Huawei SAP HANA Appliance Two
Node Installation Guide

(CH121&CH242&2288H&2488H&900
8 V5)
Issue 08
Date 2019-05-10
HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2019. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior written
consent of Huawei Technologies Co., Ltd.
Trademarks and Permissions
and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective
holders.
Notice
The purchased products, services and features are stipulated by the contract made between Huawei and the
customer. All or part of the products, services and features described in this document may not be within the
purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information,
and recommendations in this document are provided "AS IS" without warranties, guarantees or
representations of any kind, either express or implied.
The information in this document is subject to change without notice. Every effort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, express or implied.
Huawei Technologies Co., Ltd.

Address: Huawei Industrial Base
Bantian, Longgang
Shenzhen 518129
People's Republic of China
Website: http://e.huawei.com
Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. i

Huawei SAP HANA Appliance Two Node Installation
Guide (CH121&CH242&2288H&2488H&9008 V5) About This Document
About This Document
Purpose
This guide describes the installation and configuration of the two-node SAP HANA high
availability (HA) cluster solution.
You can refer to this guide to learn SAP HANA and HA cluster solutions and perform HA
cluster installation planning, operating system (OS) and database installation, network
configuration, NTP service configuration, system replication configuration, and cluster HA
configuration on the two SAP HANA server nodes.
OSs include the following:
l SUSE Linux Enterprise Server (SLES) 12 SP2/SP3/SP4, 15

l Red Hat Enterprise Linux (RHEL) 7.3/7.4
SAP HANA databases include the following:
l SAP HANA Database 2.0

l SAP HANA Database 1.0
Intended Audience
This document is intended for:
l Technical support engineers

l Maintenance engineers
l Users
Symbol Conventions
The symbols that may be found in this document are defined as follows.
Symbol Description
Indicates an imminently hazardous situation which, if not

avoided, could result in death or serious injury.
Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. ii

Symbol Description
Indicates a potentially hazardous situation which, if not

avoided, could result in death or serious injury.

avoided, could result in minor or moderate injury.

avoided, could result in equipment damage, data loss,
performance deterioration, or unanticipated results.
NOTICE is used to address practices not related to
personal injury.
Calls attention to important information, best practices

and tips.
NOTE is used to address information not related to
personal injury, equipment damage, and environment
deterioration.
Change History
Changes between document issues are cumulative. The latest document issue contains all the
changes made in earlier issues.
Issue 08 (2019-05-10)
This issue is the eighth official release.Updated 8.6 Configuring Cluster Resources and 11
OS Lifecycle.
Issue 07 (2019-03-28)
This issue is the seventh official release:
l Added the content of SLES 12 SP4, and SLES 15.

l Deleted the content of 8100 V5
l Added the content of 9008 V5
Issue 06 (2018-12-26)
This is the sixth official release. Updated Solution Overview and Installation Planning.
Issue 05 (2018-09-25)
This is the fifth official release. Added 12 Responsibility Matrix and Problem Handling
Process.
Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. iii

Issue 04 (2018-08-10)
This is the fourth official release. Updated some information in the document.
Issue 03 (2018-07-06)
This issue is the third official release. Added the description about SUSE Linux 12.3.
Issue 02 (2018-05-22)
This issue is the second official release.
Issue 01 (2018-04-03)
This issue is the first official release.
Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. iv

Guide (CH121&CH242&2288H&2488H&9008 V5) Contents
Contents
About This Document.....................................................................................................................ii

1 Solution Overview........................................................................................................................ 1
2 Installation Planning.................................................................................................................... 4
2.1 Hardware Planning......................................................................................................................................................... 4
2.2 Network Planning........................................................................................................................................................... 6
2.3 Software Planning (SLES)..............................................................................................................................................8
2.4 Software Planning (RHEL).......................................................................................................................................... 12
3 OS and Database Installation................................................................................................... 17

3.1 Installing HA Software Packages (RHEL)................................................................................................................... 17
4 Network Configuration.............................................................................................................. 19
4.1 Configuring a Blade Server (CH242 V5)..................................................................................................................... 19
4.1.1 Configuring the MM910 Management Network Out Mode..................................................................................... 19
4.1.2 Configuring Switch Module Stacking....................................................................................................................... 19
4.2 Setting IP Addresses (SLES)........................................................................................................................................ 21
4.3 Setting IP Addresses (RHEL).......................................................................................................................................27
4.4 Modifying the hosts File...............................................................................................................................................28
4.5 Configuring the SSH Password-free Interconnection Service......................................................................................29
5 NTP Service Configuration....................................................................................................... 32

5.1 Configuring an NTP Server.......................................................................................................................................... 32
5.2 Configuring an NTP Client...........................................................................................................................................33
6 CHRONY Service Configuration (SLES 15)........................................................................... 35

6.1 Configuring a CHRONY Server...................................................................................................................................35
6.2 Configuring a CHRONY Client................................................................................................................................... 36
7 System Replication Configuration...........................................................................................37

7.1 Installing the SAP HANA Client..................................................................................................................................37
7.2 Creating User Keys.......................................................................................................................................................38
7.3 Creating Database Backups.......................................................................................................................................... 40
7.4 Configuring System Replication Channels...................................................................................................................41
7.5 Configuring the Active Node....................................................................................................................................... 42
7.6 Configuring the Standby Node..................................................................................................................................... 43
Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. v

7.7 Checking the System Replication Status...................................................................................................................... 46
8 Cluster HA Configuration (SLES 12)....................................................................................... 48

8.1 Installing the Cluster HA Component.......................................................................................................................... 48
8.2 Installing the System Replication Patch Package.........................................................................................................53
8.3 Initializing the Active Node......................................................................................................................................... 54
8.4 Adding the Standby Node to the Cluster...................................................................................................................... 55
8.5 Configuring Basic Cluster Parameters......................................................................................................................... 56
8.6 Configuring Cluster Resources.....................................................................................................................................62
8.7 Configuring Cluster Resource Status........................................................................................................................... 63
8.8 Viewing Cluster Resource Status..................................................................................................................................67
8.10 HA Takeover Prerequisite...........................................................................................................................................69
8.11 Recovery Operation and Procedure After HA Takeover............................................................................................ 70
9 Cluster HA Configuration (SLES 15)....................................................................................... 73

9.1 Installing the Cluster HA Component.......................................................................................................................... 73
9.2 Installing the System Replication Patch Package.........................................................................................................78
9.3 Initializing the Active Node......................................................................................................................................... 79
9.4 Adding the Standby Node to the Cluster...................................................................................................................... 80
9.5 Configuring Basic Cluster Parameters......................................................................................................................... 81
9.6 Configuring Cluster Resources.....................................................................................................................................88
9.7 Configuring Cluster Resource Status........................................................................................................................... 89
9.8 Viewing Cluster Resource Status..................................................................................................................................92
9.10 HA Takeover Prerequisite...........................................................................................................................................95
9.11 Recovery Operation and Procedure After HA Takeover............................................................................................ 96
10 Cluster HA Configuration (RHEL).........................................................................................98

10.1 Configuring Basic Functions...................................................................................................................................... 98
10.2 Configuring Cluster Resources.................................................................................................................................102
10.3 HA Cluster Takeover Prerequisite............................................................................................................................ 107
11 OS Lifecycle.............................................................................................................................. 109
11.1 SLES for SAP Lifecycle........................................................................................................................................... 109
11.2 RHEL for SAP Lifecycle.......................................................................................................................................... 111
12 Responsibility Matrix and Problem Handling Process................................................... 113

12.1 Responsibility Matrix............................................................................................................................................... 113
12.2 Problem Handling Process........................................................................................................................................113
12.2.1 Vendor Contact Information.................................................................................................................................. 114
12.2.2 SAP Ticket Submission Process............................................................................................................................ 114
12.2.3 Huawei Ticket Submission Process....................................................................................................................... 117
12.2.4 SUSE Ticket Submission Process..........................................................................................................................117
12.2.5 Red Hat Ticket Submission Process...................................................................................................................... 120
Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. vi

13 Support and Other Resources............................................................................................... 126

13.1 Technical Support..................................................................................................................................................... 126
A Appendix 1.................................................................................................................................128
B Software Version List of Red Hat 7.4 Patch Packages in OneBox...................................131
Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. vii

Guide (CH121&CH242&2288H&2488H&9008 V5) 1 Solution Overview
1 Solution Overview
Figure 1-1 shows the typical networking of an SAP HANA two-node cluster. The two servers
with the same hardware configuration form a two-node cluster to provide services externally.
Figure 1-1 SAP HANA two-node cluster networking (CH121 V5&CH242 V5)
Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. 1

Figure 1-2 SAP HANA two-node cluster networking (2288H&2488H V5)
Figure 1-3 SAP HANA two-node cluster networking (9008 V5)

NOTE
1. The two 10GE switches need to be stacked to form an HA network.

2. On the network, you need to configure an NTP server for time synchronization in the cluster.

Guide (CH121&CH242&2288H&2488H&9008 V5) 2 Installation Planning
2 Installation Planning
2.1 Hardware Planning

2.2 Network Planning
2.3 Software Planning (SLES)
2.4 Software Planning (RHEL)
2.1 Hardware Planning

Table 2-1 Hardware planning (CH242 V5&CH121 V5&9008 V5)
Item Hardware Model Description
SAP HANA appliance Two SAP HANA Provide computing resources.

appliances
Server General-purpose server Functions as an NTP server to

synchronize time between cluster
nodes.
Switch Two 10GE switch Connect the server to the uplink

modules, for example, service network, SR network
CX320 between servers, and management
network.
FC switch Two 8 GB FC switch Connect the server to a storage

modules, for example, area network (SAN).
CX220
Cable l Power cable Connect physical devices in the

l PGND cable system.
l Ethernet cable
l Optical cable

Table 2-2 Hardware planning (2288H&2488H V5)


appliances

nodes.

S6700 between servers, and management
network.
Two 1GE switch
modules, for example,
S5728

CX220

l Ethernet cable
l Optical cable
Table 2-3 Hardware planning (9008 V5)


appliances

nodes.

CX320 between servers, and management
network.

CX220

l Ethernet cable
l Optical cable

2.2 Network Planning

Table 2-4 Network planning (CH121 V5&CH242 V5&9008 V5)
Item IP Address Remarks
Planning
MM910 192.168.34.112 Used to log in to the E9000 management

management module.
network
Switch module 192.168.34.134 Used to log in to the CX320 management

management 192.168.34.135 network.
network
BMC management Node1: Used to manage servers.

network port 192.126.126.13
Node2:
192.126.126.14
OS management Node1: It is recommended that the OS management

network port 192.126.126.131 IP address be in the same network segment
Node2: as the BMC management IP address or be
192.126.126.142 able to communicate with the BMC network
segment.
Internal cluster Node1: 10.5.5.131 Used to bond the two 10GE NICs to
network port Node2: 10.5.5.142 function as a system replication channel for
bond_sr (10GE database synchronization and function as
network port) the cluster heartbeat.
The IP address of the internal cluster
network port bond_sr cannot be in the same
IP network segment as that of the BMC
management network port or OS
management network port.
NOTE
You are advised to select one 10GE port from
each NIC and bond the two 10GE ports.


Planning
Service network port Node1: Used to bond the two 10GE NICs of the
bond_vip (10GE 126.126.126.131 server to provide services externally. The
network port) Node2: VIP must be in the same network segment
126.126.126.142 as the two nodes and is used for external
systems to access the SAP HANA.
VIP:
126.126.126.223 The IP address of the service network port
bond_vip cannot be in the same IP network
segment as that of the bond_sr, BMC
management network port, or OS
NOTE
Table 2-5 Network planning (2288H&2488H V5)

Planning
BMC management Node1: Used to manage servers.

network port 192.126.126.13
Node2:
192.126.126.14
OS management Node1: It is recommended that the OS management

network port 192.126.126.131 IP address be in the same network segment
Node2: as the BMC management IP address or be
192.126.126.142 able to communicate with the BMC network
segment.
Internal cluster Node1: 10.5.5.131 Used to bond the two 10GE NICs to
network port Node2: 10.5.5.142 function as a system replication channel for
bond_sr (10GE database synchronization and function as
network port) the cluster heartbeat.
The IP address of the internal cluster
network port bond_sr cannot be in the same
IP network segment as that of the BMC
management network port or OS
NOTE


Planning
Service network port Node1: Used to bond the two 10GE NICs of the
bond_vip (10GE 126.126.126.131 server to provide services externally. The
network port) Node2: VIP must be in the same network segment
126.126.126.142 as the two nodes and is used for external
systems to access the SAP HANA.
VIP:
126.126.126.223 The IP address of the service network port
bond_vip cannot be in the same IP network
segment as that of the bond_sr, BMC
management network port, or OS
NOTE
NOTE
For details about how to configure the IP addresses of the E9000 management modules and switch
modules, see the E9000 Server Deployment Guide and CX320 Switch Module Configuration Guide.
2.3 Software Planning (SLES)

Table 2-6 Software planning of SLES 12 SP2
Software (Version) How to obtain Description
SLES-12-SP2-DVD- https://www.suse.com/ Customized SAP OS

x86_64-GM-DVD.iso
kernel- https://www.suse.com/ SUSE Linux kernel package

default-4.4.121-92.101.1.x8
6_64.rpm

dracut-044.1-109.26.1.x86_ https://www.suse.com/ GCC 5 patches (for HANA

64.rpm 2.0)
glibc-2.22-62.10.1.x86_64.r
pm
glibc-32bit-2.22-62.10.1.x86
_64.rpm
glibc-
devel-2.22-62.10.1.x86_64.r
pm
glibc-
devel-32bit-2.22-62.10.1.x8
6_64.rpm
glibc-
i18ndata-2.22-62.10.1.noarc
h.rpm
glibc-
info-2.22-62.10.1.noarch.rp
m
glibc-
locale-2.22-62.10.1.x86_64.
rpm
glibc-
locale-32bit-2.22-62.10.1.x8
6_64.rpm
libsystemd0-32bit-228-150.
32.1.x86_64.rpm
libsystemd0-228-150.32.1.x
86_64.rpm
libudev1-32bit-228-150.32.1
.x86_64.rpm
libudev1-228-150.32.1.x86_
64.rpm
nscd-2.22-62.10.1.x86_64.r
pm
systemd-32bit-228-150.32.1
.x86_64.rpm
systemd-228-150.32.1.x86_
64.rpm
systemd-bash-
completion-228-150.32.1.no
arch.rpm
systemd-
sysvinit-228-150.32.1.x86_6
4.rpm

udev-228-150.32.1.x86_64.r
pm
IMDB_SERVER20_024_0- https:// HAHA database software

80002031.SAR launchpad.support.sap.com/
SAPCAR_816-80000935.E #/softwarecenter
XE

SLES-12-SP3-DVD- https://www.suse.com/ Customized SAP OS

x86_64-GM-DVD.iso

default-4.4.162-94.72.1.x86
_64.rpm

XE

SLE-12-SP4-SAP-DVD- https://www.suse.com/ Customized SAP OS

x86_64-GM-DVD1

default-4.12.14-95.6.1.x86_
64.rpm

XE

Table 2-9 Software planning of SLES 15

SLE-15-Installer-DVD- https://www.suse.com/ Customized SAP OS

x86_64-GM-DVD1.iso
SLE-15-Packages-x86_64-
GM-DVD1.iso

default-4.12.14-25.25.1.x86 SAPHanaSR package
_64.rpm
SAPHanaSR-0.152.22-4.3.2
.noarch.rpm
SAPHanaSR-0.152.22-4.3.2
.src.rpm
SAPHanaSR-
doc-0.152.22-4.3.2.noarch.r
pm
libopenssl1_0_0-1.0.2p-3.11
.1.x86_64.rpm

XE
NOTE
IMDB_SERVER20_020_0-80002031.SAR: SERVER20 indicates HANA 2.0, and SERVER10

indicates HANA 1.0.
Software directories on the pre-installed server are as follows:

OS destination path: /home/softwarepackage/os
Patch destination path: /home/softwarepackage/patch
HANA database destination path: /home/softwarepackage/hanadatabase
You can perform the following operations to find the OS software desired:
Step 1: Enter https://www.suse.com/ in the address box of the browser to go to the SUSE
official website.
Step 2: Click Free Downloads on the right and download the OS as required. The related
kernel can be searched and downloaded in the search box above.
NOTE
A SUSE subscription account is required for downloading a patch package.

2.4 Software Planning (RHEL)

Table 2-10 Software planning (RHEL 7.3)
Software Name and Source Description
Version
rhel-server-7.3-x86_64- https://access.redhat.com/ RHEL 7.3 OS

dvd.iso downloads/content/69/ver=/
rhel---7/7.5%20Beta/
x86_64/product-software
kernel-3.10.0-693.2.2.el7.x8 https://access.redhat.com/ RHEL kernel patch file

6_64.rpm downloads/content/package-
browser

SAPCAR_816-80000935.E #/softwarecenter/search/
XE IMDB_SERVER


Version
l compat-sap-c+ https://access.redhat.com/ RHEL 7.3 patch file

+-6-6.3.1-1.el7_3.x86_6 downloads/content/package-
4.rpm browser
l dracut-033-502.el7.x86_
64.rpm
l dracut-config-
generic-033-502.el7.x86
_64.rpm
l dracut-config-
rescue-033-502.el7.x86_
64.rpm
l dracut-
network-033-502.el7.x86
_64.rpm
l glibc-2.17-196.el7.x86_6
4.rpm and glibc-
common-2.17-196.el7.x8
6_64.rpm
l glibc-
devel-2.17-196.el7.x86_
64.rpm
l glibc-
headers-2.17-196.el7.x86
_64.rpm
l kernel-3.10.0-693.2.2.el7
.x86_64.rpm
l kexec-
tools-2.0.14-17.el7.x86_
64.rpm
l linux-
firmware-20170606-56.g
itc990aae.el7.noarch.rpm
l python-linux-
procfs-0.4.9-3.el7.noarch
.rpm
l python-
schedutils-0.4-6.el7.x86_
64.rpm
l tuned-2.8.0-5.el7.noarch.
rpm
l tuned-profiles-sap-
hana-2.8.0-5.el7.noarch.r
pm


Version
l tuned-
utils-2.8.0-5.el7.noarch.r
pm
Table 2-11 Software planning (RHEL 7.4)

Software Name Source Description How to Obtain
and Version in OneBox
rhel-server-7.4- https://access.redhat.com/ RHEL 7.4 OS https://

x86_64-dvd.iso downloads/content/69/ onebox.huawei.c
ver=/rhel---7/7.5%20Beta/ om/p/
x86_64/product-software 539042b7f82153
13f88014943a0d
4ad4
See B Software https://access.redhat.com/ RHEL 7.4 patch https://

Version List of Red downloads/content/ file onebox.huawei.c
Hat 7.4 Patch package-browser om/p/
Packages in a232cacd6f47c2
OneBox. 791d7e809c8d09
f1db
IMDB_SERVER20_ https:// HAHA database https://

020_0-80002031.SA launchpad.support.sap.co software onebox.huawei.
R m/#/softwarecenter/ com/p/
SAPCAR_816-8000 search/IMDB_SERVER 25bc6ad395e67
0935.EXE 11bd1ee6f64d44
b8ee8
https://
onebox.huawei.c
om/p/
41cc57bbfa8f14
834a077a5951a1
9ec6
To download OS software, perform the following steps:

Step 1: In the address box of the browser, type the software source link.
Step 2: Select an OS and click Download.

Step 3: To download the corresponding kernel, click the search box in the upper right corner
and enter the kernel name to find the kernel and download it.
NOTE
A SUSE subscription account is required for downloading a patch package.
Step 4: To download the corresponding RPM package, click RPM Package Search on the
right and enter the software package name, for example, dracut-033 -502.el7.x86_64.rpm.
Do not enter any space or other characters along with the software package name.


Guide (CH121&CH242&2288H&2488H&9008 V5) 3 OS and Database Installation
3 OS and Database Installation
Install the OS and database on the two SAP HANA servers. For operation details, see the
Huawei SAP HANA Appliance Single Node System Installation Guide
(CH121&CH242&2288H&2488H&9008 V5). After the installation is complete, ensure that:
l The database instance numbers of the two servers are the same.
l The database SIDs of the two servers are the same.
l The database passwords of the two servers are the same.
3.1 Installing HA Software Packages (RHEL)
3.1 Installing HA Software Packages (RHEL)

NOTE
Install an operating system (OS), and then install the database. For details, see the Huawei SAP HANA
Appliance Single Node System Installation Guide (CH121&CH242&2288H&2488H&9008 V5). After
the database is installed, install the software packages required by HA.
Step 1 Log in to server node 1.
Step 2 Mount the image on the KVM.

Step 3 Run the mount /dev/sr0 /mnt command to mount the image to the /mnt directory.
The image mounted on the KVM is generally mapped to /dev/sr0 (varied depending on the
actual situation).
[root@hw00002 dev]# mount /dev/sr0 /mnt
mount: /dev/sr0 is write-protected, mounting read-only
Step 4 Configure the yum repository.

Go to /etc/yum.repos.d/, create a yumnew.repo file (the file name can be customized), and
add the following content to the file. Then, press Esc, type :wq, and press Enter to save the
yum repository configuration.

Guide (CH121&CH242&2288H&2488H&9008 V5) 3 OS and Database Installation
[root@hw00002 etc]# cd yum.repos.d/

[root@hw00002 yum.repos.d]# vim yumnew.repo
[local-Server]
name=local-Server
baseurl=file:///mnt/
gpgcheck=0
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
keeppackages=1
[ResilientStorage]name=ResilientStorage
baseurl=file:///mnt/addons/ResilientStorage
gpgcheck=0
enabled=1
keeppackages=1
[HighAvailability]
name=HighAvailability
baseurl=file:///mnt/addons/HighAvailability
gpgcheck=0
enabled=1
keeppackages=1
Step 5 Download the following RPM packages from the official SAP website and upload them to /
home on the server:
l perl-Sys-Syslog-0.33-3.el7.x86_64
l resource-agents-3.9.5-82.el7_3.6.x86_64
l resource-agents-sap-3.9.5-82.el7_3.6.x86_64
l resource-agents-sap-hana-3.9.5-82.el7_3.6.x86_64
Step 6 Install software packages in /home.
yum install -y pacemaker corosync
yum localinstall resource-agents-sap-3.9.5-105.el7_4.2.x86_64.rpm
yum localinstall resource-agents-sap*
yum install pcs fence-agents-all
yum install gtk2 libicu xulrunner sudo tcsh libssh2 expect cairo graphviz iptraf-
ng krb5-workstation krb5-libs libpng12 nfs-utils lm_sensors rsyslog openssl
PackageKit-gtk3-module libcanberra-gtk2 libtool-ltdl xorg-x11-xauth numactl
xfsprogs net-tools bind-utils openssl098e tuned tuned-utils libtool-ltdl ntp
----End

Guide (CH121&CH242&2288H&2488H&9008 V5) 4 Network Configuration
4 Network Configuration
4.1 Configuring a Blade Server (CH242 V5)

4.2 Setting IP Addresses (SLES)
4.3 Setting IP Addresses (RHEL)
4.4 Modifying the hosts File
4.5 Configuring the SSH Password-free Interconnection Service
4.1 Configuring a Blade Server (CH242 V5)
4.1.1 Configuring the MM910 Management Network Out Mode

Step 1 Log in to the MM910 management network over SSH. (The default user name is root and the
default password is Huawei12#$.)
Step 2 Run the smmset -l smm -d outportmode -v 0 command to configure the login to the
management plane from the switch module.
----End
4.1.2 Configuring Switch Module Stacking

NOTE
Log in to the switch modules in slots 2X and 3X over SSH. (The default user name is root and the
default password is Huawei12#$.)

Step 1 Run the shutdown command to disable the stack ports.

*************SWI2 configuration*********************
<HUAWEI> system-view
[*HUAWEI]interface stack-port 2/1
[*HUAWEI-Stack-Port2/1]shutdown
[*HUAWEI-Stack-Port2/2]commit
[*HUAWEI-Stack-Port3/1]shutdown
Step 2 Configure the port stacking.

[~HUAWEI]stack
[~HUAWEI-stack]stack member 2 domain 888
[*HUAWEI-stack]port mode stack interface 40GE2/18/1
[*HUAWEI-stack]interface stack-port 2/1
[*HUAWEI-Stack-Port2/1]port member-group interface 40GE2/18/1
[~HUAWEI]stack
[~HUAWEI-stack]stack member 3 domain 888
[*HUAWEI-stack]port mode stack interface 40GE3/18/1
[*HUAWEI-stack]interface stack-port 3/1
[*HUAWEI-Stack-Port3/1]port member-group interface 40GE3/18/1
Step 3 Save the configurations.

*************Operations on SWI2*********************
<HUAWEI> save
Warning: The current configuration will be written to the device. Continue? [Y/
N]:y
<HUAWEI> save
N]:y
Step 4 Run the undo shutdown command to enable the stack ports.
[*HUAWEI-Stack-Port2/1]undo shutdown
[*HUAWEI]commit
[*HUAWEI-Stack-Port3/1]undo shutdown
[*HUAWEI]commit
Step 5 Restart the switch modules.

<HUAWEI> reboot
Warning: The system will reboot. Continue? [Y/N]:y
<HUAWEI> reboot
Warning: The system will reboot. Continue? [Y/N]:y
Step 6 Save the configurations.

*************Operations on the stack system*********************
<HUAWEI> save
N]:y

<HUAWEI>display stack
<HUAWEI>display stack configuration all
----End
4.2 Setting IP Addresses (SLES)

NOTE
1. Log in to the two server nodes as the root user and perform the following operations to set the IP
addresses.
2. The following operations use the system replication port bond_sr (physical ports eth3 and eth4 of
different NICs) as an example.
3. Each compute node has two mezzanine cards. You are advised to use one network port on each
mezzanine card and connect the ports to different switch modules to improve network redundancy.
For details about the E9000 internal networking, see A Appendix 1.
Step 2 Run yast2, in the System area, and then click Network Settings.
Figure 4-1 SLES 12

Figure 4-2 SLES 15
Step 3 Select the first 10GE network port and click Edit.
NOTE
On the switch, run the display mac command to view the mappings between ports and MAC addresses,
and obtain the MAC address of the network port to be configured. On the server, run the ifconfig
command to obtain the target Ethernet port, and configure the port bonding.
Step 4 Select No Link and IP Setup (Bonding Slaves) and click Next.

Step 5 Repeat the preceding steps to configure the second 10GE port.
Step 6 Click Add.

Step 7 Select Bond in the Device Type drop-down list, enter the name bond_sr, and click Next.
Step 8 On the Address tab page, select Statically assigned IP Address, and enter the planned IP
address and subnet mask.

Step 9 Click the Bond Slaves tab, select the two network ports (one from each NIC) to be bonded. In
Bond Driver Options, select mode=2 (or mode=balance-xor), and click Next.
NOTE
The configured bonding mode must be consistent with the bonding mode of the switch port on the
network.

Step 10 Repeat the preceding steps to configure other bond network ports. Then, click OK to save the
configuration.

Step 11 Log in to server node 2 and set IP addresses in the same way.
----End
4.3 Setting IP Addresses (RHEL)

NOTE
1. Log in to the two server nodes as the root user and perform the following operations to set the IP
addresses.
2. The following operations use the system replication port bond_sr (physical ports ens4f0 and ens6f0)
as an example.
3. If you use the same method to configure bond_vip, change the bond name and IP address in the
commands.
Step 1 Log in to server node 1 and create bond_sr and bond_vip.
Step 2 Create bond_sr.

nmcli con add type bond ifname bond_sr mode balance-xor
Step 3 Set an IP address for bond_sr.

nmcli connection modify bond-bond_sr ipv4.addresses 10.5.5.131/24
nmcli connection modify bond-bond_sr ipv4.method manual
Step 4 Add slave ports.
In this example, the slave ports are ens4f0 and ens6f0. Change them based on the actual
situation.

nmcli con add type bond-slave ifname ens4f0 master bond-bond_sr

nmcli con add type bond-slave ifname ens6f0 master bond-bond_sr
Step 5 View configuration information.

nmcli connection show
If the following information is displayed, the configuration is successful.

bond-slave-ens4f0 1c88d785-b280-4999-96d5-237d4e761e43 802-3-ethernet ens4f0
bond-slave-ens6f0 e6845549-b897-4cb8-9487-9004c71e3741 802-3-ethernet ens6f0
bond-bond_sr 5a3cb631-be4c-44d5-ae9a-95b01d6a6c03 bond
bond_sr
Step 6 Create and configure bond_vip and bond_os in the same way.
Step 7 Create the default upstream service gateway.

1. Add the gateway information to the network file, and save the modification and exit.
In the etc/sysconfig directory, run the vi network command to create and edit routes,
and add the following gateway information: The gateway IP address 10.10.1.254 is only
an example. Change it based on the actual situation.
[root@ hw00001 sysconfig] # vi network
GATEWAY=10.10.1.254
2. Run the /etc/init.d/network restart command to restart the network.

[root@ hw00001 sysconfig] # /etc/init.d/network restart
Step 8 Log in to server node 2 and set IP addresses in the same way.
----End
4.4 Modifying the hosts File

IP address and host name resolution must contain three network segments of the appliance:
l Service network segment: use the server host name defined by the customer.
l Directly connected network segment for HANA system replication (SR): use the format
of "Host name + SR".
l OS system management network segment: use "host name + management gateway".
NOTE
Log in to the two server nodes as the root user and modify the hosts file to enable host name resolution
between the two server nodes.
Step 2 Run the vi /etc/hosts command to open the hosts file.

vi /etc/hosts
Step 3 Press i to make the hosts file editable and add the following information to the file.
The host names and IP addresses in the following text are examples only and must be changed
as required.
10.5.5.142 hw00002SR
10.5.5.131 hw00001SR
192.126.126.131 hw00001
192.126.126.142 hw00002
126.126.126.142 hw00002MG
126.126.126.131 hw00001MG

Step 4 Press Esc to switch the vi editor to the CLI mode. Press the colon (:) key to switch to the
bottom line mode. Type wq and press Enter to save the modification and exit the vi editor.
Step 5 Log in to server node 2 and repeat the preceding steps to edit the hosts file.
10.5.5.142 hw00002SR
10.5.5.131 hw00001SR
192.126.126.131 hw00001
192.126.126.142 hw00002
126.126.126.142 hw00002MG
126.126.126.131 hw00001MG
----End
4.5 Configuring the SSH Password-free Interconnection

Service
NOTE
Log in to the two server nodes as the root user and perform the following operations to enable the SSH
password-free interconnection service for the two service nodes.
Step 1 Run the ssh-keygen -t rsa and ssh-keygen -t dsa commands to generate public keys for
authentication.
Server node 1
hw00001: # ssh-keygen -t rsa
hw00001: # ssh-keygen -t dsa
Run the ssh-keygen -t rsa command. The command output is as follows:

hw00001: # ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): //Press Enter to retain
the default value.
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): //Press Enter to retain the default
value.
Enter same passphrase again: //Press Enter to retain the default value.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
b9:67:bb:07:06:8b:20:1b:3a:a4:2e:ed:e3:d7:58:54 [MD5] root@hw00001
The key's randomart image is:
+--[ RSA 2048]----+
| |
| E |
| . |
| .o . . .. |
|o. + o .So |
|+ . o ..o |
|.o + ..o. |
|..o o . o .. |
|.ooo oo |
+--[MD5]----------+
Run the ssh-keygen -t rsa command. The command output is as follows:

hw00001:~ # ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): //Press Enter to retain
the default value.
Enter passphrase (empty for no passphrase): //Press Enter to retain the default
value.

Enter same passphrase again: //Press Enter to retain the default value.
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
SHA256:8DeBxFJO5TwveFcvnzN8PpjVnxO5wScZugg9Om+bhz0 root@hw00001
The key's randomart image is:
+---[DSA 1024]----+
| o+.. |
| .+.+ |
| ..o = . |
| o . + ... |
| S.= o.oo+|
| .oo+. +O+|
| o = .+*X|
| o +.Eo *=|
| ++o . +|
+----[SHA256]-----+
Run the following commands on server node 2:

ssh-keygen -t rsa
ssh-keygen -t dsa
Step 2 Copy the public key for authentication from the local node to the peer node.
NOTE
Before performing this operation, you must enable the host name and IP address resolution in the /etc/
hosts file.
hw00001 and hw00002 are examples only. Change them based on the actual situation.
Server node 1
ssh hw00001 "echo $(cat /root/.ssh/id_dsa.pub) >> /root/.ssh/authorized_keys"
ssh hw00002 "echo $(cat /root/.ssh/id_dsa.pub) >>/root/.ssh/authorized_keys"
ssh hw00001 "echo $(cat /root/.ssh/id_rsa.pub) >> /root/.ssh/authorized_keys"
ssh hw00002 "echo $(cat /root/.ssh/id_rsa.pub) >>/root/.ssh/authorized_keys"
Enter the password of the root user when prompted, as shown in the following example.
hw00001: # ssh hw00001 "echo $(cat /root/.ssh/id_dsa.pub) >> /root/.ssh/
authorized_keys"
The authenticity of host 'hw00001 (192.168.10.100)' can't be established.
ECDSA key fingerprint is ee:4c:78:4b:d8:5f:8d:44:85:c5:46:9c:90:9d:13:bd [MD5].
Are you sure you want to continue connecting (yes/no)? //Enter yes.
Warning: Permanently added 'hw00001,192.168.10.100' (ECDSA) to the list of known
hosts.
Password: //Enter the password of user root.
Server node 2
ssh hw00001 "echo $(cat /root/.ssh/id_dsa.pub) >> /root/.ssh/authorized_keys"
ssh hw00002 "echo $(cat /root/.ssh/id_dsa.pub) >>/root/.ssh/authorized_keys"
ssh hw00001 "echo $(cat /root/.ssh/id_rsa.pub) >> /root/.ssh/authorized_keys"
ssh hw00002 "echo $(cat /root/.ssh/id_rsa.pub) >>/root/.ssh/authorized_keys"
Step 3 Verify the configuration.
On the two server nodes, use SSH to log in to each other. If the logins are successful without
entering the password, the trust relationship is established.
l On server node 1, run the ssh hw00002 command to log in to service node 2 without
entering the password.

l On server node 2, run the ssh hw00001 command to log in to service node 1 without
entering the password.
----End

Guide (CH121&CH242&2288H&2488H&9008 V5) 5 NTP Service Configuration
5 NTP Service Configuration
Before installing and configuring cluster HA, configure the NTP service on the two SAP
HANA server nodes and synchronize time between the two server nodes.
5.1 Configuring an NTP Server
5.2 Configuring an NTP Client
5.1 Configuring an NTP Server

NOTE
l This section uses a Linux OS as an example to describe how to configure the NTP server. If an NTP
server exists on the live network, skip this section and configure the NTP client.
l Before configuring NTP, set the time zone in the OS to the local time zone. This document uses the
time zone of China Shanghai as an example. On the customer site, use the actual time zone.
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
Step 1 Log in to the NTP server as the root user. Right-click in the blank space on the desktop and
choose Open Terminal from the shortcut menu.
Step 2 Run the systemctl enable ntpd.service command to configure automatic startup for the NTP
service.
# systemctl enable ntpd.service
Step 3 Run the vi /etc/ntp.conf command to open the NTP configuration file.
# vi /etc/ntp.conf
Step 4 Press i to make the ntp.conf file editable and add NTP client IP address restriction to the file.
Format:
server 127.127.1.0
fudge 127.127.1.0 stratum 10
restrict <NTP client IP address> mask <NTP client subnet mask> nomodify notrap
This command allows an NTP client with the specified IP address to use the current host as
the NTP server and synchronize time from this host.
Example:
restrict 192.126.126.0 mask 255.255.255.0 nomodify

This example command allows NTP clients with the IP addresses 192.126.126.1–
192.126.126.254 to synchronize time from the NTP server.
In this command, set this parameter to the IP address of the OS management network port.
Step 6 Run the systemctl restart ntpd.service command to restart the NTP service.
# systemctl restart ntpd.service
----End
5.2 Configuring an NTP Client

NOTE
Log in to the two server nodes as the root user and perform the following operations.
Step 2 Run the systemctl enable ntpd.service command to configure automatic startup for the NTP
service.
# systemctl enable ntpd.service
Step 3 Run the vi /etc/ntp.conf command to open the NTP configuration file.
# vi /etc/ntp.conf
Step 4 Press i to make the ntp.conf file editable and add NTP server IP address to the file.
Format:
server <NTP server IP address>
The following uses 192.126.126.131 as an example. Replace it with the actual NTP server IP
address.
server 192.126.126.131
Step 6 Run the systemctl restart ntpd.service command to restart the NTP service.
# systemctl restart ntpd.service
Step 7 Run the ntpq -p command to check the NTP running status.
hw00001: # ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
192.126.126.131 .LOCL. 1 u 2 64 1 0.417 5.373 0.000
Table 5-1 Parameter description
Parameter Description
remote Indicates the name of the NTP server that responds to the
request.

when Indicates the number of seconds since the last successful

request.
poll Indicates the interval (in seconds) for the local host to
synchronize time between the local NTP client and the remote
NTP server.
offset Indicates the time offset (in milliseconds) between the local
NTP client and the time source.
NOTE
If the time difference (offset) is greater than 1000 seconds, you need to
stop the ntpd service, manually synchronize the time, and start the ntpd
service.
systemctl stop ntpd.service
ntpdate 192.126.126.131 //NTP server IP address.
Replace it with the actual NTP server IP address.
systemctl start ntpd.service
----End

Guide (CH121&CH242&2288H&2488H&9008 V5) 6 CHRONY Service Configuration (SLES 15)
6 CHRONY Service Configuration (SLES 15)
During Cluster HA installation and configuration on SLES 15, CHRONY is used as the
default time synchronization tool for the SAP HANA server. If the customer can provide a
CHRONY server, perform steps in this chapter to configure CHRONY time synchronization.
Otherwise, use the NTP tool for time synchronization.
6.1 Configuring a CHRONY Server
6.2 Configuring a CHRONY Client
6.1 Configuring a CHRONY Server

NOTE
This section uses a Linux OS as an example to describe how to configure the CHRONY server. If a
CHRONY server exists on the live network, skip this section and configure the CHRONY client.
Step 1 Log in to the CHRONY server as the root user. Right-click in the blank space on the desktop
and choose Open Terminal from the shortcut menu.
Step 2 Run the systemctl enable chronyd.service command to configure automatic startup for the
CHRONY service.
# systemctl enable chronyd.service
Step 3 Run the vi /etc/chrony.conf command to open the CHRONY configuration file.
# vi /etc/chrony.conf
Step 4 Press i to make the ntp.conf file editable and add client IP address restriction to the file.
Format:
server 127.0.0.1 iburst
allow <CHRONY client IP address> /<CHRONY lient subnet mask>
This command allows an CHRONY client with the specified IP address to use the current
host as the CHRONY server and synchronize time from this host.
Example:
allow 192.126.126.0/16
This example command allows CHRONY clients with the IP addresses 192.126.126.1–
192.126.126.254 to synchronize time from the CHRONY server.

Guide (CH121&CH242&2288H&2488H&9008 V5) 6 CHRONY Service Configuration (SLES 15)
In this command, set this parameter to the IP address of the OS management network port.
Step 6 Run the systemctl restart chronyd.service command to restart the CHRONY service.
# systemctl restart chronyd.service
----End
6.2 Configuring a CHRONY Client

NOTE
Log in to the two server nodes as the root user and perform the following operations.
Step 2 Run the systemctl enable chronyd.service command to configure automatic startup for the
CHRONY service.
# systemctl enable chronyd.service
Step 3 Run the vi /etc/chrony.conf command to open the CHRONY configuration file.
# vi /etc/chrony.conf
Step 4 Type i to enter the edit mode, add the IP address of the CHRONY server to the file, comment
out IP addresses of other servers, and write the following command to the file:
Format:
server < CHRONYP server IP address> iburst
The following uses 192.126.126.131 as an example. Replace it with the actual CHRONY
server IP address.
server 192.126.126.131 iburst
Step 6 Run the systemctl restart chronyd.service command to restart the CHRONY service.
# systemctl restart chronyd.service
----End

Guide (CH121&CH242&2288H&2488H&9008 V5) 7 System Replication Configuration
7 System Replication Configuration
7.1 Installing the SAP HANA Client

7.2 Creating User Keys
7.3 Creating Database Backups
7.4 Configuring System Replication Channels
7.5 Configuring the Active Node
7.6 Configuring the Standby Node
7.7 Checking the System Replication Status
7.1 Installing the SAP HANA Client

NOTE
l The database software version must match the client version. The SAP_HANA_Client version
described in this section is used as an example.
l There are two installation methods depending on the installation source obtaining method of the
SAP HANA Client.
l Log in to the two server nodes as the root user and perform the operations described in this section.
Installation Method 1
Step 1 Download the SAP HANA Client installation package (x86_64), for example,
IMDB_CLIENT20_002_36-80002082.SAR, and the decompression file
SAPCAR_816-80000935.exe, from the official SAP website.
Step 2 Upload the installation package and decompression software to a directory, such as /home, on
the SAP HANA server.
Step 4 Run the chmod +x * command to add the execute permission.
hw00001:/home# chmod +x *
Step 5 Run the ./SAPCAR_816-80000935.exe -xvf IMDB_CLIENT20_002_36-80002082.SAR

command to decompress the installation package. The SAP_HANA_CLIENT directory is
generated after the decompression.

hw00001:/home # ./SAPCAR_816-80000935.exe -xvf IMDB_CLIENT20_002_36-80002082.SAR
Step 6 Run the cd SAP_HANA_CLIENT command to go to the SAP_HANA_CLIENT directory.

hw00001:/home # cd SAP_HANA_CLIENT
Step 7 Run the ./hdbinst command to install the SAP HANA Client.
Retain the default installation directory /usr/sap/hdbclient when prompted. If hdblcm_gui is

used, the default installation directory is /hana/shared/<SID>/hdbclient.
hw00001:/home /SAP_HANA_CLIENT # ./hdbinst
Step 8 Log in to server node 2 and install the SAP HANA Client in the same way.
----End
Installation Method 2
Download the SAP HANA Client installation package from the official SAP website.
Decompress it to the local PC and upload the HDB_CLIENT_LINUX_X86_64 installation
package generated after the decompression to the SAP HANA server nodes. The procedure is
as follows:
Step 1 Download the SAP HANA Client installation package (x86_64), for example,
IMDB_CLIENT20_002_36-80002082.SAR, from the official SAP website.
Step 2 Decompress the package on the local computer to obtain the SAP HANA Client installation
package HDB_CLIENT_LINUX_X86_64 (The package name depends on the actual
situation).
Step 3 Upload HDB_CLIENT_LINUX_X86_64 to the SAP HANA server nodes.
Step 5 Run the chmod +x * command to add the execute permission.

hw00001:/home# chmod +x *
Step 6 Run the cd ./SAP_HANA_CLIENT command to go to the SAP_HANA_CLIENT directory.

hw00001:/home # cd ./SAP_HANA_CLIENT
Step 7 Run the ./hdbinst command to install the SAP HANA Client.
Retain the default installation directory /usr/sap/hdbclient when prompted. If hdblcm_gui is

used, the default installation directory is /hana/shared/<SID>/hdbclient.
hw00001:/home /SAP_HANA_CLIENT # ./hdbinst
Step 8 Log in to server node 2 and install the SAP HANA Client in the same way.
----End
7.2 Creating User Keys

NOTE
l Skip this section if the SLES system is used.

l For an RHEL system, start the database, log in to the two server nodes as the root user, and perform
the following operations:

Step 2 Go to the hdbclient directory (this section uses usr/sap/hdbclient as an example).

cd /usr/sap/hdbclient
Step 3 In the usr/sap/hdbclient # directory, run the following command to create a user:
./hdbsql -i 00 -u system -p <SYTEM password> -n localhost:30015 "create user
rhelhasync password <password>"
30015 Indicates the port number. 00 indicates the database instance

ID in this example. 15 indicates the port number in a single-
tenant scenario. The port number will be 13 in a multi-tenant
scenario. In the multi-tenant scenario, when the instance ID is
01, the value is 30113.
Step 4 Assign permissions to the new user rhelhasync.

./hdbsql -i 00 -u system -p <SYTEM password> -n localhost:30015 "grant CATALOG
READ to rhelhasync"
./hdbsql -i 00 -u system -p <SYTEM password> -n localhost:30015 "grant MONITOR
ADMIN to rhelhasync"
./hdbsql -i 00 -u system -p <SYTEM password> -n localhost:30015 "ALTER USER
rhelhasync DISABLE PASSWORD LIFETIME"
Step 5 Create the user key for the rhelhasync user to log in to the database.
./hdbuserstore SET SAPHANAS00SR localhost:30015 rhelhasync <password>
password Indicates the password of the rhelhasync user. Replace it with

the actual password.
30015 Indicates the port number. 00 indicates the database instance

ID in this example. 15 indicates the port number in a single-
tenant scenario. The port number will be 13 in a multi-tenant
scenario. In the multi-tenant scenario, when the instance ID is
01, the value is 30113.
Step 6 Run the ./hdbuserstore list command to check whether the user key is created successfully.
If ENV is 30015 (or 30013 in the multi-tenant scenario) and USER is the name of the newly
created user, the user key is created successfully.
[root@hw00001 hdbclient]# ./hdbuserstore list
DATA FILE : /root/.hdb/hw00001/SSFS_HDB.DAT
KEY FILE : /root/.hdb/hw00001/SSFS_HDB.KEY
KEY SAPHANAS00SR
ENV : localhost:30015
USER: rhelhasync
Step 7 Run the ./hdbsql -U SAPHANAS00SR "select * from dummy" command to check whether
the user key is correct.

NOTE
Before you run the command, ensure that the database is running in primary mode. The database is
supposed not to be registered with another database. Otherwise, the command execution fails.
If DUMMY X is displayed, the password is correct. Otherwise, the password is incorrect.

[root@hw00001 hdbclient]#./hdbsql -U SAPHANAS00SR "select * from dummy"
DUMMY
"X"
Step 8 Log in to server node 2 and create a user key in the same way.
----End
7.3 Creating Database Backups

NOTE
Before you perform the following operations, ensure that the database is running in primary mode. The
database is supposed not to be registered with another database. Otherwise, the command execution
fails. Log in to the two server nodes as the root user and use the SAP HANA studio management tool or
command-line interface (CLI) to perform the following operations.
Step 2 Run the cd /usr/sap/hdbclient command to go to the /usr/sap/hdbclient directory.

hw00001:/ # cd /usr/sap/hdbclient
Step 3 Create a database backup.

l Single-tenant scenario:
Run the ./hdbsql -i 00 -u SYSTEM -p Huawei_123 -n localhost:30015 "BACKUP
DATA USING FILE ('backup')" command to create a database backup.
If the following message is displayed, the database backup is created.
hw00001:/usr/sap/hdbclient # ./hdbsql -i 00 -u SYSTEM -p Huawei_123 -n
localhost:30015 "BACKUP DATA USING FILE ('backup')"
0 rows affected (overall time 7332.151 msec; server time 7330.445 msec)

30015 Indicates the port number. 00 indicates the database

instance ID in this example. 15 indicates the port number in
a single-tenant scenario. The port number will be 13 in a
multi-tenant scenario. In the multi-tenant scenario, when
the instance ID is 01, the value is 30113.
l Multi-tenant scenario:
Run the following commands to back up the SYSEMDB and tenantDB:
NOTE
If the database version is HANA 2.0 or multi-tenant deployment (multiple_containers) is selected

during the database installation, you need to back up SYSEMDB and tenantDB. Huawei12#$ is
the database password. In 30013, 00 is the database instance number and S00 is the database SID.
Change them based on the actual situation.
./hdbsql -i 00 -u SYSTEM -p Huawei12#$ -n localhost:30013 "backup data for
S00 using file ('COMPLETE_DATA_BACKUP')"

./hdbsql -i 00 -u SYSTEM -p Huawei12#$ -n localhost:30013 "backup data using

file ('COMPLETE_DATA_BACKUP')"
Step 4 Log in to server node 2 and create a database backup in the same way.
----End
7.4 Configuring System Replication Channels

NOTE
Log in to the two server nodes as the root user and perform the operations described in this section.
Step 2 Run the su - s00adm command to switch to the database user.
s00adm is the database user name automatically generated during the database installation,
and s00 is the database SID in lowercase. Change them based on the actual situation.
hw00001:/ # su - s00adm
Step 3 Run the HDB stop command to stop the database.

hw00001:/usr/sap/S00/HDB00> HDB stop
Step 4 Run the exit command to the switch to the root user.
hw00001:/usr/sap/S00/HDB00> exit
hw00001:/ #
Step 5 Run the vim /hana/shared/S00/global/hdb/custom/config/global.ini command to open the

global.ini file.
The SID must be changed based on the actual situation. In this example, the SID is S00.
hw00001:/ # vim /hana/shared/S00/global/hdb/custom/config/global.ini
Step 6 Press i to make the hosts file editable and add the following information to the file.
10.5.5.142 is the IP address of the dedicated system replication 10GE port of the peer server,
and hw00002 is the host name of the peer server. Change them based on the actual situation.
[system_replication_communication]
listeninterface = .global
[system_replication_hostname_resolution]
10.5.5.142 = hw00002
bottom line mode. Type :wq and press Enter to save the modification and exit the vi editor.
Step 8 Log in to server node 2, and edit the global. ini file in the same way. (The peer IP address and
host name need to be modified accordingly.)
[system_replication_communication]
listeninterface = .global
[system_replication_hostname_resolution]
10.5.5.131 = hw00001
----End

7.5 Configuring the Active Node

In this document, server node 1 is the active node. Enable the SAP HANA system replication
function on the active node. Before enabling the system replication function, ensure that the
network connection is normal and the HANA database is running.
Step 1 Log in to server node 1 as the root user.

hw00001:/ # su - s00adm
Step 3 Run the HDB start command to start the database.

s00adm@hw00001:/usr/sap/S00/HDB00> HDB start
StartService
Impromptu CCC initialization by 'rscpCInit'.
See SAP note 1266393.
OKOK
Starting instance using: /usr/sap/S00/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00
-function StartWait 2700 2
12.10.2015 15:05:28
StartOK
12.10.2015 15:06:26
StartWaitOK
Step 4 Run the hdbnsutil -sr_enable --name=hw00001 command to enable system replication.
name=hw00001 is the local host name. Change it based on the actual situation.
s00adm@hw00001:/usr/sap/S00/HDB00> hdbnsutil -sr_enable --name=hw00001
checking for active nameserver ...
nameserver is active, proceeding ...
successfully enabled system as system replication source site
done.
Step 5 Run the hdbnsutil -sr_state command to check the database system replication status.
The current database mode is primary, which indicates that the local node is the active node.
s00adm@hw00001:/usr/sap/S00/HDB00> hdbnsutil -sr_state
checking for active or inactive nameserver ...
System Replication State
~~~~~~~~~~~~~~~~~~~~~~~
online: true
mode: primary
operation mode: primary
site id: 1
site name: hw00001
is source system: true

is secondary/consumer system: false
has secondaries/consumers attached: false
is a takeover active: false
Host Mappings:
~~~~~~~~~~~~~~
Site Mappings:
~~~~~~~~~~~~~~

hw00001 (primary/)
Tier of hw00001: 1
Replication mode of hw00001: primary
Operation mode of hw00001:
done.
----End
7.6 Configuring the Standby Node

In this document, server node 2 is the standby node. Enable the SAP HANA system
replication function on the standby node and register the standby node with the active node.
Before enabling the system replication function, ensure that the network connection is normal
and the HANA database is stopped. (Start the HANA database after enabling the system
replication function.)
Step 1 Log in to server node 2 as the root user.

hw00002:/ # su - s00adm
Step 3 Run the HDB stop command to stop the database on the standby node.
hw00002:/usr/sap/S00/HDB00> HDB stop
Stopping instance using: /usr/sap/S00/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00
-function Stop 400
12.10.2015 15:11:23
Stop
OK
Waiting for stopped instance using: /usr/sap/S00/SYS/exe/hdb/sapcontrol -prot
NI_HTTP -nr 00 -function WaitforStopped 600 2
12.10.2015 15:11:23
WaitforStopped
OK
hdbdaemon is stopped.
Step 4 Copy the SSFS_<SID>.DAT and SSFS_<SID>.KEY files from the active node to the
standby node.
In SAP HANA 2.0, the data and log transmission channel needs to be authenticated during the
system replication process. Therefore, the system PKI SSFS storage certificate is required.
SSFS_<SID>.DAT and SSFS_<SID>.KEY are respectively stored in /usr/sap/SID/SYS/
global/security/rsecssfs/data and /usr/sap/SID/SYS/global/security/rsecssfs/key. SID is the
database instance ID. In this example, the SID is S00. Replace it with the actual SID.
1. On the standby node, run the following commands to back up the files:
[root@hw00002 ~]# cd /usr/sap/S00/SYS/global/security/rsecssfs/data
[root@hw00002 data]# mv SSFS_S00.DAT SSFS_S00.DAT.bak
[root@hw00002 ~]# cd /usr/sap/S00/SYS/global/security/rsecssfs/key
[root@hw00002 key]# mv SSFS_S00.KEY SSFS_S00.KEY.bak
2. On the active node, run the following commands to synchronize files to the standby
node:
hw00001:cd /usr/sap/S00/SYS/global/security/rsecssfs/key
hw00001:/usr/sap/S00/SYS/global/security/rsecssfs/key # scp SSFS_S00.KEY
s00adm@hw00002:/usr/sap/S00/SYS/global/security/rsecssfs/key

Password: //Enter the password.

SSFS_S00.KEY
100% 187 0.2KB/s
00:00
hw00001:cd /usr/sap/S00/SYS/global/security/rsecssfs/data
hw00001:/usr/sap/S00/SYS/global/security/rsecssfs/data # scp SSFS_S00.DAT
s00adm@hw00002:/usr/sap/S00/SYS/global/security/rsecssfs/data
Password: //Enter the password.
SSFS_S00.DAT
100% 2960 2.9KB/s
00:00
Step 5 On the standby node, run the hdbnsutil -sr_register --remoteHost=hw00001 --

remoteInstance=00 --replicationMode=sync --name=hw00002 --
operationMode=logreplay command to register the standby node with the active node.
s00adm@hw00002:/usr/sap/S00/HDB00> hdbnsutil -sr_register --remoteHost=hw00001 --
remoteInstance=00 --replicationMode=sync --name=hw00002 --operationMode=logreplay
adding site ...
checking for inactive nameserver ...
nameserver hw00002:30001 not responding.
collecting information ...
registered at 192.126.126.131 (hw00001)
updating local ini files ...
done.
s00adm@hw00002:/usr/sap/S00/HDB00>

remoteHost Indicates the host name of the active node. In this example, the
host name is hw00001. Replace it with the actual host name.
name Indicates the host name of the standby node. In this example,
the host name is hw00002. Replace it with the actual host
name.
replicationMode Indicates the synchronization mode, including:

l sync: hard disk synchronization
l async: asynchronization
l syncmem: memory synchronization
NOTICE
You are advised to select sync. If you select async or sysnmem,
performance is improved but there are data loss risks when exceptions
occur.
remoteInstance Indicates the HANA database instance ID. Replace it with the
actual instance ID.

operationMode For HANA1.0:

l delta_datashipping: The data is periodically transmitted
between the active and standby nodes and the standby node
spends a long time to take over the database service.
delta_datashipping is the default mode.
l logreplay: After the data on the active node is initialized
and synchronized to the standby node, only log files are
synchronized. In this mode, the logs of the active node are
redone on the standby node in real time and the standby
node spends a short time to take over the database service.
logreplay is recommended.
For HANA2.0:
l delta_datashipping: The data is periodically transmitted
between the active and standby nodes and the standby node
spends a long time to take over the database service.
l logreplay: After the data on the active node is initialized
and synchronized to the standby node, only log files are
synchronized. In this mode, the logs of the active node are
redone on the standby node in real time and the standby
node spends a short time to take over the database service.
The default mode is logreplay (recommended).
l logreplay_readaccess: After the data on the active node is
initialized and synchronized to the standby node, only log
files are synchronized. In this mode, the logs of the active
node are redone on the standby node in real time and the
standby node spends a short time to take over the database
service. In addition, the SQL can access the standby node in
read-only mode.
Step 6 Run the HDB start command to start the database on the standby node.
s00adm@hw00002:/usr/sap/S00/HDB00> HDB start
StartService
Impromptu CCC initialization by 'rscpCInit'.
See SAP note 1266393.
OK
OK
Starting instance using: /usr/sap/S00/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00
-function StartWait 2700 2
12.10.2015 15:16:53
Start
OK
12.10.2015 15:17:27
StartWait
OK
Step 7 Run the hdbnsutil -sr_state command to check the system replication status on the standby
node.
If the database mode is sync and two nodes can be found, the synchronization mode is
registered successfully.
s00adm@hw00002:/usr/sap/S00/HDB00> hdbnsutil -sr_state

System Replication State

~~~~~~~~~~~~~~~~~~~~~~~~
online: true
mode: sync
operation mode: logreplay
site id: 2
site name: hw00002
is source system: false

is secondary/consumer system: true
has secondaries/consumers attached: false
is a takeover active: false
active primary site: 1
primary masters: hw00001
Host Mappings:
~~~~~~~~~~~~~~
hw00002 -> [hw00001] hw00001

hw00002 -> [hw00002] hw00002
Site Mappings:
~~~~~~~~~~~~~~
hw00001 (primary/primary)
|---hw00002 (sync/logreplay)
Tier of hw00001: 1
Tier of hw00002: 2
Replication mode of hw00001: primary

Replication mode of hw00002: sync
Operation mode of hw00001: primary

Operation mode of hw00002: logreplay
Mapping: hw00001 -> hw00002

done.
----End

The system replication duration required by the HANA database depends on the data volume
of the HANA database and the network bandwidth of the system synchronization channel.
Step 1 Log in to the active node as the root user. Run the su - s00adm command to switch to the
HANA database user. Replace s00 in the command with the actual database SID in lowercase.
Step 2 Run the cdpy command as a database user to go to the python directory.
Step 3 On the active node, run the python systemReplicationStatus.py command using a database
account to check the data synchronization status.
If the displayed status is ACTIVE, the database has completed synchronization and remains
in sync mode, and you can perform the database takeover operation.
If the status is Initializing, the database is synchronizing data and is not ready for the
takeover operation.
The following is an example of the command output:

s00adm@hw00001:/usr/sap/POC/HDB00> cdpy
s00adm@hw00001:/usr/sap/S00/HDB00/exe/python_support> python
systemReplicationStatus.py
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name |
Secondary | Secondary | Secondary | Secondary | Secondary | Replication |
Replication | Replication |
| | | | | | | |
Host | Port | Site ID | Site Name | Active Status | Mode |
Status | Status Details |
| -------- | ---- | ----- | ------------ | --------- | ------- | --------- |
--------- | --------- | --------- | --------- | ------------- | ----------- |
----------- | -------------- |
| SYSTEMDB | hw00001 | 30001 | nameserver | 1 | 1 | hw00001
| hw00002 | 30001 | 2 | hw00002 | YES |
SYNC | ACTIVE | |
| TD1 | hw00001 | 30040 | indexserver | 2 | 1 | hw00001
| hw00002 | 30040 | 2 | hw00002 | YES |
SYNC | ACTIVE | |
status system replication site "2": ACTIVE

overall system replication status: ACTIVE
Local System Replication State

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mode: PRIMARY
site id: 1
site name: hw00001
s00adm@hw00001:/usr/sap/S00/HDB00/exe/python_support>
You can also use HANA studio to connect to the HANA database on the active node and
check the system replication status on the Overview tab page. As shown in the following
figure, if the icon is green (All service are active and in sync), the synchronization is
complete; if the icon is yellow, the synchronization is still being implemented.
----End

Guide (CH121&CH242&2288H&2488H&9008 V5) 8 Cluster HA Configuration (SLES 12)
8 Cluster HA Configuration (SLES 12)
8.1 Installing the Cluster HA Component

8.2 Installing the System Replication Patch Package
8.3 Initializing the Active Node
8.4 Adding the Standby Node to the Cluster
8.5 Configuring Basic Cluster Parameters
8.6 Configuring Cluster Resources
8.7 Configuring Cluster Resource Status
8.8 Viewing Cluster Resource Status
8.10 HA Takeover Prerequisite
8.11 Recovery Operation and Procedure After HA Takeover

NOTE
l This section describes how to install and configure the SLES 12 SP3 for SAP. The procedure for
installing SLES 12 SP3 for SAP is similar.
l Log in to the active and standby nodes of the SAP HANA database as the root user and perform the
following operations.
Step 1 Log in to the active node.
Step 2 Mount the SLES 12 SP3 For SAP ISO file.
Step 3 Right-click the desktop and open a terminal. On the terminal, run the yast2 command to open
the YaST Control Center.

Step 4 On the YaST Control Center, choose Software > Add-On Products.
Step 5 Click Add.

Step 6 Select DVD.

Step 7 Insert the add-on product DVD and click Continue.
Step 8 Select I Agree to the License Terms and click Next.
Step 9 Click the Patterns tab, select High Availability, and click Accept.

Step 10 Click Continue to finish the installation.

Step 11 Log in to the standby node and install the cluster HA component in the same way.
----End

NOTE
1. If the OS is RHEL, skip this section.

2. If the OS is SLES, log in to the active and standby nodes of the SAP HANA server as the root user.
(You can specify the active and standby nodes based on the site requirements or use the NTP server
as the active node.) Perform the following operations.
Step 1 Download the SAP HANA System Replication patch package from the official website based
on the OS version. The following patch packages are for reference only.
l SAPHanaSR-0.152.22-1.1.noarch.rpm
l SAPHanaSR-0.152.22-1.1.src.rpm
l SAPHanaSR-doc-0.152.22-1.1.noarch.rpm
Step 2 Upload the patch packages to any directories on the active and standby nodes.
Step 3 Log in to the active node and go to the directory where the patch packages are stored.
Step 4 Run the rpm -Uvh *.rpm command to install the patch packages.
rpm -Uvh *.rpm
Step 5 Log in to the standby node and install the patch packages in the same way.
----End


NOTE
Before initialization, ensure that you have configured SSH mutual trust and NTP.
Step 1 Log in to the active node as the root user.

Step 2 Add the default route. In the command, eth0 is the OS network port of the active node. Set the
actual OS network port according to the actual situation, and then run the /sbin/ip route
command.
hw00001:#ip route add default via 192.126.126.131 dev eth0
hw00001:/home # /sbin/ip route
Step 3 Run the ha-cluster-init command to initialize the active node.

hw00001:# ha-cluster-init
WARNING: No watchdog device found. If SBD is used, the cluster will be unable to
start without a watchdog.
Do you want to continue anyway? [y/N] y //Enter y to use IPMI instead of SBD as
the split-brain mechanism.
Command line is not complete. Try option "help"
WARNING: Could not detect IP address for
WARNING: Could not detect network address for Enabling sshd service
/root/.ssh/id_rsa already exists - overwrite? [y/N] n// Enter n, which indicates
not to overwrite id_rsa.
Configuring csync2
csync2 is already configured - overwrite? [y/N] y //Enter y, which indicates that
the service has been configured before and overwrites the original configuration.
The message is not displayed in SLES 12 SP1 for SAP.
Generating csync2 shared key (this may take a while)...done
Enabling csync2 service
Enabling xinetd service
csync2 checking files
Configure Corosync:
This will configure the cluster messaging layer. You will need
to specify a network address over which to communicate (default
is eth0's network, but you can use the network address of any
active interface), a multicast address and multicast port.
Network address to bind to (e.g.: 192.168.1.0) []192.168.1.0 //Enter the OS
maintenance network segment of the local host.
Multicast address (e.g.: 239.x.x.x) [239.80.241.100] //This IP address is not
changed by default. Record the IP address because it will be used in the basic
cluster parameter configuration.
Multicast port [5405] //Press Enter to retain the value.
Configure SBD:
If you have shared storage, for example a SAN or iSCSI target,
you can use it avoid split-brain scenarios by configuring SBD.
This requires a 1 MB partition, accessible to all nodes in the
cluster. The device path must be persistent and consistent
across all nodes in the cluster, so /dev/disk/by-id/* devices
are a good choice. Note that all data on the partition you
specify here will be destroyed.
Do you wish to use SBD? [y/N] n // Enter n, which indicates that the SBD mode is
not used. To use the SBD mode, enter y.
WARNING: Not configuring SBD - STONITH will be disabled.
Enabling hawk service
HA Web Konsole is now running, to see cluster status go to:
https://192.168.20.210:7630/
Log in with username 'hacluster', password 'linux' //Remember that the login user
is hacluster and the password is linux.
WARNING: You should change the hacluster password to something more secure!
Enabling openais service
Waiting for cluster........done
Loading initial configuration
Configure Administration IP Address:

Optionally configure an administration virtual IP

address. The purpose of this IP address is to
provide a single IP that can be used to interact
with the cluster, rather than using the IP address
of any specific cluster node.
Do you wish to configure an administration IP? [y/N] n //Enter n to finish the

initialization without configuring the management IP address.
Done (log saved to /var/log/ha-cluster-bootstrap.log)
----End

Step 1 Log in to the active node as the root user and run the scp /etc/corosync/corosync.conf
hw00002:/etc/corosync/ command to copy the corosync configuration file from the active
node to the standby node.
In this example, hw00002 is the host name of the standby node. Replace it with the actual
host name or IP address of the standby node.
scp /etc/corosync/corosync.conf hw00002:/etc/corosync/
Step 2 Log in to the standby node of the cluster as the root user, and run the following commands to
add the default route. In the commands, eth2 is the OS management network port of the
standby node. Change it based on the site requirements.
hw00002 :~ # ip route add default via 192.126.126.142 dev eth2
hw00002 :~ # /sbin/ip route
Step 3 Log in to the standby node as the root user and run the ha-cluster-join command to add the
standby node to the cluster.
hw00002:~ # ha-cluster-join
WARNING: Could not detect network address for eth0
Join This Node to Cluster:
You will be asked for the IP address of an existing node, from which
configuration will be copied. If you have not already configured
passwordless ssh between nodes, you will be prompted for the root
password of the existing node.
IP address or hostname of existing node (e.g.: 192.168.1.1) [] 192.168.1.143 //
Enter the OS management IP address of the active node.
Enabling sshd service
/root/.ssh/id_rsa already exists - overwrite? [y/N] n//Enter n, which indicates
/root/.ssh/id_dsa already exists - overwrite? [y/N] n//Enter n, which indicates
not to overwrite id_dsa.
Configuring csync2
Merging known_hosts
Probing for new partitions......done
https://10.5.2.11:7630/
Log in with username 'hacluster', password 'linux'
Waiting for cluster...done
Done (log saved to /var/log/sleha-bootstrap.log)
hw00002:~ #

NOTE
If the message "WARNING: csync2 of /etc/csync2/csync2.cfg failed - file may not be in sync on all
nodes;" is displayed, the installation fails. Perform the following operations:
1. On the active node, run the scp /etc/csync2/csync2.cfg hw00002:/etc/csync2/ command to manually
synchronize the csync2.cfg file from the active node to the standby node. In the command, hw00002
is the host name of the standby node. Replace it with the actual host name or IP address of the
standby node.
2. On the standby node, run the ha-cluster-join command.
----End

Step 2 Type yast2 in the command line, and select Cluster.
Step 3 Select Communication Channels to set the parameters, and click Finish.
1. Set Transport to Unicast.
2. Set Channel to the active heartbeat channels.
– Bind Network Address: Enter the OS management network segment, for example,
192.126.126.0.
– Port: Retain the default value 5405.
3. Select Redundant Channel to configure the standby heartbeat channel.
– Bind Network Address: Enter the network segment for direct connections with the
system replication databases, for example, 10.5.5.0.

– Port: Enter 5407.

4. Configure Member Address. Click Add to add the IP addresses of the OS management
network ports of the two servers in the cluster (192.126.126.131 and 192.126.126.142 in
this example) and the IP addresses for the system replication direct connections
(10.5.5.131 and 10.5.5.142 in this example).
5. Configure Cluster Name.
6. Retain the default value of Expected Votes.
7. Set rrp mode to passive.
8. Select Auto Generate Node ID.
Step 4 Choose Security, select Enable Security Auth, and click Generate Auth Key File.
Step 5 Choose Configure Csync2, click Generate Pre-Shared-Keys to generate the key, and click
Add Suggested Files to add files to be synchronized. Select hw00002 in the Sync Host list
box, and delete /etc/multipath.conf in the Sync File list box. Delete the same file for
hw00001. (If the /etc/multipath.conf file is synchronized, the server will enter the emergency
mode.)

NOTE
Ensure that Turn csync2 OFF is displayed, which indicates that csync2 is enabled.
Step 6 Choose Configure conntrackd, set the parameters as shown in Table 8-1, and click
Generate *. conf.


Dedicated Interface Select the OS management network port. In this example, the
value is eth0:192.126.126.131.
Multicast Address Enter the default multicast address 239.80.241.100, which is

generated during the active node initialization.
NOTE
If there are two HA environments, two IP addresses are generated
during the active node initialization. Use different multicast addresses,
for example, 239.80.241.110.
Group Number Enter 1.
Step 7 Choose Service, set the parameters as shown in Table 8-2, and click Finish.

Booting On is selected by default, which indicates that the pacemaker

cluster is automatically started when the server is started.
NOTE
To use the manual cluster startup mode, set Booting to Off for both
nodes.
Switch On and Off Set this parameter to Start pacemaker Now, which indicates
that the pacemaker is enabled.
Firewall Settings Ensure that the firewall is disabled.
Step 8 On the active node, run the scp /etc/csync2/key_hagroup /etc/csync2/csync2.cfg

hw00002:/etc/csync2 command to manually synchronize the Shared-key file.

Manually synchronize the shared key file after configuring basic cluster parameters. Copy
the /etc/csync2/key_hagroup and /etc/csync2/csync2.cfg files from the active node to
the /etc/csync2 directory on the standby node. hw00002 is the host name of the standby node.
Replace it with the actual host name or IP address of the standby node.
scp /etc/csync2/key_hagroup /etc/csync2/csync2.cfg hw00002:/etc/csync2
Step 9 On the active node, run the following commands to enable the csync2 and xinetd functions:
systemctl enable csync2.socket
systemctl enable xinetd
systemctl restart xinetd
Step 10 On the standby node, run the following commands to enable the csync2 and xinetd functions:
systemctl enable xinetd
systemctl restart xinetd
Step 11 On the active node, run the csync2 -xv command to synchronize the active and standby
configuration files. Check the multipath file multipath.conf in the /etc directory on the
standby node. If the multipath file of the active node is synchronized to the standby node,
reconfigure the multipath file of the standby node according to the Huawei SAP HANA
Appliance Single Node System Installation Guide (CH121&CH242&2288H&2488H&9008
V5). Otherwise, the standby node will restart and enter the maintenance mode due to
multipath issues.
csync2 -xv
NOTE
If the following information is displayed, the synchronization fails:

"While syncing file /etc/corosync/corosync.conf:
ERROR from peer hex-14: File is also marked dirty here!
Finished with 1 errors."
Run the following commands to forcibly synchronize files:
csync2 -f /etc/corosync/corosync.conf
csync2 -x
Step 12 On the active node, run the systemctl status pacemaker and systemctl restart pacemaker
commands to restart the cluster service.
systemctl status pacemaker
systemctl restart pacemaker
Step 13 On the standby node, run the systemctl status pacemaker and systemctl restart pacemaker
Step 14 On the active node, run the crm_mon -r command to check the cluster status. The active and
standby nodes are online.
hw00001:~ # crm_mon -r
Stack: corosync
Current DC: hw00001 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Wed May 9 11:20:09 2018
Last change: Wed May 9 11:19:42 2018 by root via crm_attribute on hw00001
2 nodes configured
7 resources configured
Online: [ hw00001 hw00002 ]
Step 15 On the active node, run the corosync-cfgtool -s command to view the cluster heartbeat status.
The active and standby heartbeats exist.

NOTE
If the cluster is configured for the first time, there may be only one heartbeat cable (ring ID). In this
case, restart both servers and check the heartbeat status again.
hw00001:~ # corosync-cfgtool -s
Printing ring status.
Local node ID 1084752650
RING ID 0
id= 10.5.5.131
status= ring 0 active with no faults
RING ID 1
id= 192.126.126.131
Step 16 Log in to the iBMC, choose Configuration > Local Users, and grant the IPMI LAN login
interface permission to the Administrator user. The permission has been granted if the icon
is green in the IPMI column.
If the permission is not granted, click the edit icon in the Operation column and select IPMI.
Step 17 Run the ipmitool -I lanplus -H 192.126.126.14 -U Administrator -P Admin@9000 chassis

power status command to check the iBMC IPMI link status of the cluster.
On the active and standby nodes, use IPMItool to connect to the iBMCs of the active and
standby nodes respectively, and check the power status. If Chassis Power is on is displayed,
the IPMI connection is working properly.
Active node:
hw00001:~ # ipmitool -I lanplus -H 192.126.126.14 -U Administrator -P Admin@9000
chassis power status
Chassis Power is on
Standby node:


Chassis Power is on

-H 192.126.126.14 Indicates the iBMC IP address of the peer node. Change it

based on the actual situation.
-U Administrator Indicates the user name for logging in to the iBMC. Change it
-P Admin@9000 Indicates the password for logging in to the iBMC. Change it

NOTE
If the message "Get Device ID command failed: 0xc1 Invalid command" is displayed, ignore it.
----End

Step 1 Log in to the active node as the root user, and create the crm-stonith.txt and crm-stonith-
cs.txt files.
The content of the crm-stonith.txt file is as follows:

# enter the following to crm-stonith.txt
primitive rsc_hana01_stonith stonith:external/ipmi \
meta target-role=Started \
operations $id=rsc_hana01_stonith-operations \
op monitor interval=1800 timeout=20 \
params ipaddr=192.126.126.13 hostname=hw00001 userid=Administrator
passwd="Admin@9000" interface=lanplus

rsc_hana01_stonith corresponds to the configuration of node 1. Set ipaddr to the IP address

of the BMC of node 1, userid to the BMC user, and passwd to the password of the BMC user.
Set rsc_hana02_stonith that corresponds to the configuration of node 2 in the same way.
The content of the crm-stonith-cs.txt file is as follows: (Change hw00001 and hw00002 to
the actual host names.)
# enter the following to crm-stonith-cs.txt
location loc_hana01_stonith rsc_hana01_stonith -inf: hw00001
Step 2 Run the following commands to import the configuration files to the cluster.
crm configure load update crm-stonith.txt
crm configure load update crm-stonith-cs.txt

Step 3 On the active node, create the crm-bs.txt configuration file.
The content of the crm-bs.txt file is as follows:

# enter the following to crm-bs.txt
property $id="cib-bootstrap-options" \
no-quorum-policy="ignore" \
stonith-enabled="true" \
stonith-action="reboot" \
stonith-timeout="150s"
rsc_defaults $id="rsc-options" \
resource-stickiness="1000" \
migration-threshold="5000"
op_defaults $id="op-options" \
timeout="600"
Step 4 Run the crm configure load update crm-bs.txt command to import the configuration file to
the cluster.
crm configure load update crm-bs.txt
----End

Step 1 In the address box of the browser, enter https://HOSTNAME_OR_IP_ADDRESS:7630/ to
access the Hawk cluster management login page, and enter the username and password. (The
default user is hacluster, and the default password is linux.)
In the address, HOSTNAME_OR_IP_ADDRESS indicates the OS management network

port host name or IP address of any cluster node. Change it based on the actual situation.
NOTE
If a certificate warning is displayed when you attempt to access the URL for the first time, it indicates
that a self-signed certificate is used. By default, the self-signed certificate is not considered as a trusted
certificate. Click Continue to this website (not recommended) or add an exception in the browser to
eliminate the warning message.
The Hawk cluster management page is displayed.

Step 2 Choose Wizards > SAP > SAP HANA SR Scale-Up Performance-Optimized.
Step 3 Set the database SID, instance number, and virtual IP address, and click Verify.

Step 4 Verify the configuration of the SAP HANA database resource and click Apply.

NOTE
In SAPHanaSR-0.152.21-1.1, VIP resources are bound to nic=eth0, which conflicts with the solution
networking plan bond_vip. You need to manually delete the nic=eth0.
Modification method:
Log in to any cluster node as the root user and run the crm config edit command.
Delete nic=eth0 from primitive rsc_ip_S00_HDB00.

NOTE
In the preceding configuration, the database automatic takeover function is enabled. That is, if the active
node database is faulty when data has been synchronized between the active and standby nodes, the
standby node database can automatically take over the services. The automatic registration function is
disabled. That is, after the standby node takes over the services, the original active node, upon a restart,
cannot automatically register to the original standby node. After rectifying the fault on the original
active node, run the following command to register the original active node to the original standby node:
hw00001 and hw00002 are the host names of the active and standby nodes. Replace them with the
actual host names.
hdbnsutil -sr_register --remoteHost=hw00002 --remoteInstance=00 --
replicationMode=sync --name=hw00001 --operationMode=logreplay
----End

Step 1 Log in to Hawk and choose Status in the navigation tree to check the cluster status. Green
icons indicate that the cluster resources are running properly.
Step 2 Check the stonith resource status.

The stonith resource needs to run on the peer node to reboot the peer node when cluster brain
split occurs.
In this example, the stonith resource of hana01 must run on hw0002, and the stonith resource
of hana02 must run on hw0001

Step 3 Check the status of the virtual IP address.
The virtual IP address is used for the SAP application server to access the database and must
be bound to the master database node. If the virtual IP address is not bound to the master
database node, the cluster status is abnormal.
You can also run the crm_mon -r command to check the cluster status.
Stack: corosync
Current DC: hw00001 (version 1.1.15-19.15-e174ec8) - partition with quorum
Last updated: Wed Sep 6 09:58:52 2017
Last change: Wed Sep 6 09:58:47 2017 by root via crm_attribute on hw00001
2 nodes configured
Online: [ hw00001 hw00002 ]
Full list of resources:
rsc_hana01_stonith (stonith:external/ipmi): Started hw00002

rsc_ip_S00_HDB00 (ocf::heartbeat:IPaddr2): Started hw00001
Master/Slave Set: msl_SAPHana_S00_HDB00 [rsc_SAPHana_S00_HDB00]
Masters: [ hw00001 ]
Slaves: [ hw00002 ]

Clone Set: cln_SAPHanaTopology_S00_HDB00 [rsc_SAPHanaTopology_S00_HDB00]

Started: [ hw00001 hw00002 ]
Step 4 Run the crm configure command to switch to the CLI mode.
crm configure
Step 5 Run the following commands to add network monitoring resources for the upstream service
port:
In this example, 126.126.126.254 is the gateway of the service port and rsc_ip_S00_HDB00
is the name of the virtual IP. Change them based on the actual situation.
crm(live)configure# primitive r_ping ocf:pacemaker:ping params multiplier=100
dampen=5 name=pingdtest host_list=126.126.126.254 op monitor interval=15
timeout=60 op start interval=0 timeout=60
crm(live)configure# clone r_ping-clone r_ping
crm(live)configure# location loc_r_ping rsc_ip_S00_HDB00 rule -inf: not_defined
pingdtest or pingdtest lte 0 //Add constraint
Step 6 Run the commit command to submit the configuration.

crm(live)configure# commit
----End

When the active node is faulty, the standby node will take over services only after the data is
synchronized between the active and standby nodes.
Step 1 Run the SAPHanaSR-showAttr command to check the System Replication active/standby
status.
hw00001:/home # SAPHanaSR-showAttr
Host \ Attr clone_state remoteHost roles site srmode
sync_state vhost lpa_sle_lpt
----------------------------------------------------------------------------------
------------------------------
hw00001 PROMOTED hw00002 4:P:master1:master:worker:master hw00001
sync PRIM hw00001 1416991408
hw00002 DEMOTED hw00001 4:S:master1:master:worker:master hw00002
sync SOK hw00002 30
sync_state has the following types:

l PRIM indicates that the node is the active node.
l SOK indicates that the node is ready and synchronization is complete.
l SFAIL indicates that synchronization is not complete.
----End

The standby node of the cluster takes over the database only when data synchronization
between the standby node and the active node is complete.
If the data is not synchronized or the database on the active node is faulty, the standby node
does not take over the database from the active node.
Step 1 Run the SAPHanaSR-showAttr command to check the active/standby status of the SAP
HANA system replication.

Run the SAPHanaSR-showAttr command on the active node.

The result description is as follows:
sync_state:
PRIM indicates that the node is the active node.
SOK indicates that the node is ready and synchronization is complete.
SFAIL indicates that synchronization is not complete.
The command output is similar to the following:
HW00001:/home # SAPHanaSR-showAttr
Host \ Attr clone_state remoteHost roles site srmode sync_state
vhost lpa_sle_lpt
----------------------------------------------------------------------------------
-----
HW00001 PROMOTED HW00002 4:P:master1:master:worker:master HW00001 sync PRIM
HW00001 1416991408
HW00002 DEMOTED HW00001 4:S:master1:master:worker:master HW00002 sync SOK
HW00002 30
Step 2 If data synchronization is not complete, automatic HA cluster takeover will not occur. For
details about manual takeover, see the SAP notes. When the synchronization status of all
services is not Active, SAP does not recommend the takeover operation because this means
that data may be lost.
a. SAP note: 2578019 - Service Crashes in
DataAccess::PersistenceManagerImpl::endOfDataRecovery
Do not perform a takeover if not for all services the REPLICATION_STATUS in
M_SERVICE_REPLICATION shows ACTIVE. See SAP Note 2063657 for details.
https://launchpad.support.sap.com/#/notes/2578019
b. SAP note: 2580302 - Emergency Shutdown of Indexserver Due to Log Position
Inconsistency Upon Takeover
With the fix the takeover in this case can succeed, but since one service wasn't in sync this
implies data loss. You should follow the takeover decision guide of SAP Note 2063657 to
assess if a takeover in this state is a feasible option
c. SAP note: 2063657 - SAP HANA System Replication Takeover Decision Guideline
You are advised to determine whether to perform a takeover based on the site requirements.
----End
8.11 Recovery Operation and Procedure After HA

Takeover
By default, the automatic registration function is disabled. After an HA takeover is performed,
the original active node needs to be registered with the current active node. You need to
manually run the registration command. For details about the registration procedure, see Step

1 to Step 2. After the registration is complete, clear the failure count of the resource to enable
the resource to run again.
NOTE
The SLES 12 updates the failure counting mechanism. After an HA takeover, the database needs to be
registered and SAP HANA resources are cleaned up again.
Step 1 Assume that the original active node is HW00001 and the original standby node is
HW00002. After HW00002 takeover is complete, virtual IP resources are migrated to
HW00002.
Step 2 Run the hdbnsutil -sr_register --remoteHost=hw00002 --remoteInstance=00 --
replicationMode=sync --name=hw00001 --operationMode=logreplay command using the
hw00001 database account of the original active node to register hw00001 as the standby
node with the HW00002.
Step 3 Log in to SUSE Hawk, choose Cleanup in the Operations column of the SAP HANA
resource record to clear the resource failure count.
Step 4 After the failure count is cleaned up, the database automatically starts on HW00001.

----End

9 Cluster HA Configuration (SLES 15)

9.11 Recovery Operation and Procedure After HA Takeover

NOTE
Log in to the active and standby nodes of the SAP HANA database as the root user and perform the
following operations.
Step 1 Log in to the active node.
Step 2 Mount the SLES 15 For SAP ISO file.
Step 3 Right-click the desktop and click Activities to open a terminal. On the terminal, run the yast2
command to open the YaST Control Center.

Step 4 On the YaST Control Center, choose Software > Add-On Products.
Step 5 Click Add.

Step 6 Select DVD.

Step 7 Click Next. Insert the add-on product DVD and click Continue.
Step 8 Click Next. In the displayed window, click Continue.

Step 9 Click the Patterns tab, select High Availability, and click Accept.

Step 10 Click Continue to finish the installation.
Step 11 Log in to the standby node and install the cluster HA component in the same way.
----End

NOTE
If the OS is SLES, log in to the active and standby nodes of the SAP HANA server as the root user.
(You can specify the active and standby nodes based on the site requirements or use the NTP server and
CHRONY server as the active node.) Perform the following operations.
Step 1 Download the SAP HANA System Replication patch package from the official website based
on the OS version. The following patch packages are for reference only.
l SAPHanaSR-0.152.22-4.3.2noarch.rpm
l SAPHanaSR-0.152.22-4.3.2.src.rpm
l SAPHanaSR-doc-0.152.22-4.3.2.noarch.rpm
Step 2 Upload the patch packages to any directories on the active and standby nodes.
Step 3 Open the YaST2 interface and check whether crmsh and crmsh-script are installed.

Step 4 Log in to the active node and go to the directory where the patch packages are stored.
Step 5 Run the rpm -Uvh *.rpm command to install the patch packages.
rpm -Uvh *.rpm
Step 6 Log in to the standby node and install the patch packages in the same way.
----End

NOTE
Before initialization, ensure that you have configured SSH mutual trust ,NTP and CHRONY.
Step 2 Add the default route. In the command, eth0 is the OS network port of the active node. Set the
actual OS network port according to the actual situation, and then run the /sbin/ip route
command.
hw00001:#ip route add default via 192.168.2.171 dev eth6
hw00001:/home # /sbin/ip route
Step 3 On SLES 15, if the NTP time synchronization tool is used, delete the CHRONY service
automatically installed when Cluster HA is installed. Skip this step on SLES 12.
mv /usr/lib/systemd/system/chronyd.service /home
Step 4 Run the ha-cluster-init command to initialize the active node.

hw00001:# ha-cluster-init


Command line is not complete. Try option "help"
WARNING: Could not detect IP address for
WARNING: Could not detect network address for Enabling sshd service
/root/.ssh/id_rsa already exists - overwrite? [y/N] n// Enter n, which indicates
Configuring csync2
csync2 is already configured - overwrite? [y/N] y //Enter y, which indicates that
the service has been configured before and overwrites the original configuration.
Generating csync2 shared key (this may take a while)...done
csync2 checking files
Configure Corosync:
This will configure the cluster messaging layer. You will need
to specify a network address over which to communicate (default
is eth0's network, but you can use the network address of any
active interface), a multicast address and multicast port.
Network address to bind to (e.g.: 192.168.1.0) []192.168.1.0 //Enter the OS
maintenance network segment of the local host.
Multicast address (e.g.: 239.x.x.x) [239.80.241.100] //This IP address is not
changed by default. Record the IP address because it will be used in the basic
cluster parameter configuration.
Multicast port [5405] //Press Enter to retain the value.
Configure SBD:
If you have shared storage, for example a SAN or iSCSI target,
you can use it avoid split-brain scenarios by configuring SBD.
This requires a 1 MB partition, accessible to all nodes in the
cluster. The device path must be persistent and consistent
across all nodes in the cluster, so /dev/disk/by-id/* devices
are a good choice. Note that all data on the partition you
specify here will be destroyed.
Do you wish to use SBD? [y/N] n // Enter n, which indicates that the SBD mode is
not used. To use the SBD mode, enter y.
WARNING: Not configuring SBD - STONITH will be disabled.
https://192.168.20.210:7630/
Log in with username 'hacluster', password 'linux' //Remember that the login user
is hacluster and the password is linux.
Waiting for cluster........done
Loading initial configuration
Configure Administration IP Address:
Optionally configure an administration virtual IP
address. The purpose of this IP address is to
provide a single IP that can be used to interact
with the cluster, rather than using the IP address
of any specific cluster node.
Do you wish to configure an administration IP? [y/N] n //Enter n to finish the

initialization without configuring the management IP address.
Done (log saved to /var/log/ha-cluster-bootstrap.log)
----End

Step 1 Log in to the active node as the root user and run the scp /etc/corosync/corosync.conf
hw00002:/etc/corosync/ command to copy the corosync configuration file from the active
node to the standby node.

In this example, hw00002 is the host name of the standby node. Replace it with the actual
host name or IP address of the standby node.
scp /etc/corosync/corosync.conf hw00002:/etc/corosync/
Step 2 Log in to the standby node of the cluster as the root user, and run the following commands to
add the default route. In the commands, eth2 is the OS management network port of the
standby node. Change it based on the site requirements.
hw00002 :~ # ip route add default via 192.126.126.142 dev eth2
hw00002 :~ # /sbin/ip route
Step 3 Log in to the standby node as the root user and run the ha-cluster-join command to add the
standby node to the cluster.
hw00002:~ # ha-cluster-join
WARNING: Could not detect network address for eth0
Join This Node to Cluster:
You will be asked for the IP address of an existing node, from which
configuration will be copied. If you have not already configured
passwordless ssh between nodes, you will be prompted for the root
password of the existing node.
IP address or hostname of existing node (e.g.: 192.168.1.1) [] 192.168.1.143 //
Enter the OS management IP address of the active node.
Enabling sshd service
/root/.ssh/id_rsa already exists - overwrite? [y/N] n//Enter n, which indicates
/root/.ssh/id_dsa already exists - overwrite? [y/N] n//Enter n, which indicates
not to overwrite id_dsa.
Configuring csync2
Merging known_hosts
Probing for new partitions......done
https://10.5.2.11:7630/
Log in with username 'hacluster', password 'linux'
Waiting for cluster...done
Done (log saved to /var/log/sleha-bootstrap.log)
hw00002:~ #
NOTE
If the message "WARNING: csync2 of /etc/csync2/csync2.cfg failed - file may not be in sync on all
nodes;" is displayed, the installation fails. Perform the following operations:
1. On the active node, run the scp /etc/csync2/csync2.cfg hw00002:/etc/csync2/ command to manually
synchronize the csync2.cfg file from the active node to the standby node. In the command, hw00002
is the host name of the standby node. Replace it with the actual host name or IP address of the
standby node.
2. On the standby node, run the ha-cluster-join command.
----End

Step 2 Type yast2 in the command line, and select Cluster.

Step 3 Select Communication Channels to set the parameters, and click Finish.
1. Set Channel to the active heartbeat channels.
– Bind Network Address: Enter the OS management network segment, for example,
192.126.126.0.
– Port: Retain the default value 5405.
2. Select Redundant Channel to configure the standby heartbeat channel.
– Bind Network Address: Enter the network segment for direct connections with the
system replication databases, for example, 10.5.5.0.
– Port: Enter 5407.
3. Set Transport to Unicast.
4. Configure Member Address. Click Add to add the IP addresses of the OS management
network ports of the two servers in the cluster (192.126.126.131 and 192.126.126.142 in
this example) and the IP addresses for the system replication direct connections
(10.5.5.131 and 10.5.5.142 in this example).
5. Configure Cluster Name.
6. Retain the default value of Expected Votes.
7. Set rrp mode to passive.
8. Select Auto Generate Node ID.

Step 4 Choose Security, select Enable Security Auth, and click Generate Auth Key File.
Step 5 Choose Configure Csync2, click Generate Pre-Shared-Keys to generate the key, and click
Add Suggested Files to add files to be synchronized. Select hw00002 in the Sync Host list
box, and delete /etc/multipath.conf in the Sync File list box. Delete the same file for
hw00001. (If the /etc/multipath.conf file is synchronized, the server will enter the emergency
mode.)
NOTE
Ensure that Turn csync2 OFF is displayed, which indicates that csync2 is enabled.

Step 6 Choose Configure conntrackd, set the parameters as shown in Table 9-1, and click
Generate *. conf.

Dedicated Interface Select the OS management network port. In this example, the
value is eth0:192.126.126.131.

Multicast Address Enter the default multicast address 239.80.241.100, which is

generated during the active node initialization.
NOTE
If there are two HA environments, two IP addresses are generated
during the active node initialization. Use different multicast addresses,
for example, 239.80.241.110.
Group Number Enter 1.
Step 7 Choose Service, set the parameters as shown in Table 9-2, and click Finish.

Booting On is selected by default, which indicates that the pacemaker

cluster is automatically started when the server is started.
NOTE
To use the manual cluster startup mode, set Booting to Off for both
nodes.
Switch On and Off Set this parameter to Start pacemaker Now, which indicates
that the pacemaker is enabled.
Firewall Settings Ensure that the firewall is disabled.
Step 8 On the active node, run the scp /etc/csync2/key_hagroup /etc/csync2/csync2.cfg

hw00002:/etc/csync2 command to manually synchronize the Shared-key file.
Manually synchronize the shared key file after configuring basic cluster parameters. Copy
the /etc/csync2/key_hagroup and /etc/csync2/csync2.cfg files from the active node to

the /etc/csync2 directory on the standby node. hw00002 is the host name of the standby node.
Replace it with the actual host name or IP address of the standby node.
scp /etc/csync2/key_hagroup /etc/csync2/csync2.cfg hw00002:/etc/csync2
Step 9 On the active node, run the following commands to enable the csync2:
Step 10 On the standby node, run the following commands to enable the csync2:
Step 11 On the active node, run the csync2 -xv command to synchronize the active and standby
configuration files. Check the multipath file multipath.conf in the /etc directory on the
standby node. If the multipath file of the active node is synchronized to the standby node,
reconfigure the multipath file of the standby node according to the Huawei SAP HANA
Appliance Single Node System Installation Guide (CH121&CH242&2288H&2488H&9008
V5). Otherwise, the standby node will restart and enter the maintenance mode due to
multipath issues.
csync2 -xv
NOTE
If the following information is displayed, the synchronization fails:

"While syncing file /etc/corosync/corosync.conf:
ERROR from peer hex-14: File is also marked dirty here!
Finished with 1 errors."
Run the following commands to forcibly synchronize files:
csync2 -f /etc/corosync/corosync.conf
csync2 -x
Step 12 On the active node, run the systemctl status pacemaker and systemctl restart pacemaker
Step 13 On the standby node, run the systemctl status pacemaker and systemctl restart pacemaker
Step 14 On the active node, run the crm_mon -r command to check the cluster status. The active and
standby nodes are online.
hw00001:~ # crm_mon -r
Stack: corosync
Current DC: hw00001 (version 1.1.18+20180430.b12c320f5-1.14-b12c320f5) -
partition with quorum
Last updated: Wed Feb 13 15:14:27 2019
Last change: Wed Feb 13 14:49:36 2019 by hacluster via cibadmin on hw00001
2 nodes configured
Online: [ hw00001 hw00002
No resources
Step 15 On the active node, run the corosync-cfgtool -s command to view the cluster heartbeat status.
The active and standby heartbeats exist.
NOTE
If the cluster is configured for the first time, there may be only one heartbeat cable (ring ID). In this
case, restart both servers and check the heartbeat status again.

hw00001:~ # corosync-cfgtool -s
Local node ID 1084752650
RING ID 0
id= 10.5.5.131
RING ID 1
id= 192.126.126.131
Step 16 Log in to the iBMC, choose Configuration > Local Users, and grant the IPMI LAN login
interface permission to the Administrator user. The permission has been granted if the icon
is green in the IPMI column.
If the permission is not granted, click the edit icon in the Operation column and select IPMI.
Step 17 Run the ipmitool -I lanplus -H 192.126.126.14 -U Administrator -P Admin@9000 chassis

power status command to check the iBMC IPMI link status of the cluster.
On the active and standby nodes, use IPMItool to connect to the iBMCs of the active and
standby nodes respectively, and check the power status. If Chassis Power is on is displayed,
the IPMI connection is working properly.
Active node:
Chassis Power is on
Standby node:
Chassis Power is on


-H 192.126.126.14 Indicates the iBMC IP address of the peer node. Change it

-U Administrator Indicates the user name for logging in to the iBMC. Change it
-P Admin@9000 Indicates the password for logging in to the iBMC. Change it

NOTE
If the message "Get Device ID command failed: 0xc1 Invalid command" is displayed, ignore it.
----End

Step 1 Log in to the active node as the root user, and create the crm-stonith.txt and crm-stonith-
cs.txt files.
The content of the crm-stonith.txt file is as follows:
# enter the following to crm-stonith.txt

rsc_hana01_stonith corresponds to the configuration of node 1. Set ipaddr to the IP address

of the BMC of node 1, userid to the BMC user, and passwd to the password of the BMC user.
Set rsc_hana02_stonith that corresponds to the configuration of node 2 in the same way.
The content of the crm-stonith-cs.txt file is as follows: (Change hw00001 and hw00002 to
the actual host names.)
# enter the following to crm-stonith-cs.txt
Step 2 Run the following commands to import the configuration files to the cluster.
crm configure load update crm-stonith.txt
crm configure load update crm-stonith-cs.txt
Step 3 On the active node, create the crm-bs.txt configuration file.

The content of the crm-bs.txt file is as follows:
# enter the following to crm-bs.txt
property $id="cib-bootstrap-options" \

no-quorum-policy="ignore" \
stonith-enabled="true" \
stonith-action="reboot" \
stonith-timeout="150s"
rsc_defaults $id="rsc-options" \
resource-stickiness="1000" \
migration-threshold="5000"
op_defaults $id="op-options" \
timeout="600"
Step 4 Run the crm configure load update crm-bs.txt command to import the configuration file to
the cluster.
crm configure load update crm-bs.txt
----End

Step 1 In the address box of the browser, enter https://HOSTNAME_OR_IP_ADDRESS:7630/ to
access the Hawk cluster management login page, and enter the username and password. (The
default user is hacluster, and the default password is linux.)
In the address, HOSTNAME_OR_IP_ADDRESS indicates the OS management network
port host name or IP address of any cluster node. Change it based on the actual situation.
NOTE
If a certificate warning is displayed when you attempt to access the URL for the first time, it indicates
that a self-signed certificate is used. By default, the self-signed certificate is not considered as a trusted
certificate. Click Continue to this website (not recommended) or add an exception in the browser to
eliminate the warning message.

The Hawk cluster management page is displayed.
Step 2 Choose Wizards > SAP > SAP HANA SR Scale-Up Performance-Optimized.
Step 3 Set the database SID, instance number, and virtual IP address, and click Verify.

Step 4 Verify the configuration of the SAP HANA database resource and click Apply.

NOTE
In the preceding configuration, the database automatic takeover function is enabled. That is, if the active
node database is faulty when data has been synchronized between the active and standby nodes, the
standby node database can automatically take over the services. The automatic registration function is
disabled. That is, after the standby node takes over the services, the original active node, upon a restart,
cannot automatically register to the original standby node. After rectifying the fault on the original
active node, run the following command to register the original active node to the original standby node:
hw00001 and hw00002 are the host names of the active and standby nodes. Replace them with the
actual host names.
hdbnsutil -sr_register --remoteHost=hw00002 --remoteInstance=00 --
replicationMode=sync --name=hw00001 --operationMode=logreplay
----End

Step 1 Log in to Hawk and choose Status in the navigation tree to check the cluster status. Green
icons indicate that the cluster resources are running properly.

Step 2 Check the stonith resource status.
The stonith resource needs to run on the peer node to reboot the peer node when cluster brain
split occurs.
In this example, the stonith resource of hana01 must run on hw0002, and the stonith resource
of hana02 must run on hw0001

Step 3 Check the status of the virtual IP address.
The virtual IP address is used for the SAP application server to access the database and must
be bound to the master database node. If the virtual IP address is not bound to the master
database node, the cluster status is abnormal.
You can also run the crm_mon -r command to check the cluster status.
Stack: corosync
Current DC: hw00001 (version 1.1.15-19.15-e174ec8) - partition with quorum
Last updated: Wed Sep 6 09:58:52 2017
Last change: Wed Sep 6 09:58:47 2017 by root via crm_attribute on hw00001
2 nodes configured
Online: [ hw00001 hw00002 ]

rsc_ip_S00_HDB00 (ocf::heartbeat:IPaddr2): Started hw00001
Master/Slave Set: msl_SAPHana_S00_HDB00 [rsc_SAPHana_S00_HDB00]
Slaves: [ hw00002 ]
Clone Set: cln_SAPHanaTopology_S00_HDB00 [rsc_SAPHanaTopology_S00_HDB00]
Started: [ hw00001 hw00002 ]
Step 4 Run the crm configure command to switch to the CLI mode.
crm configure
Step 5 Run the following commands to add network monitoring resources for the upstream service
port:
In this example, 126.126.126.254 is the gateway of the service port and rsc_ip_S00_HDB00
is the name of the virtual IP. Change them based on the actual situation.
crm(live)configure# primitive r_ping ocf:pacemaker:ping params multiplier=100
dampen=5 name=pingdtest host_list=126.126.126.254 op monitor interval=15
timeout=60 op start interval=0 timeout=60
crm(live)configure# clone r_ping-clone r_ping
crm(live)configure# location loc_r_ping rsc_ip_S00_HDB00 rule -inf: not_defined
pingdtest or pingdtest lte 0 //Add constraint
Step 6 Run the commit command to submit the configuration.

crm(live)configure# commit
----End

When the active node is faulty, the standby node will take over services only after the data is
synchronized between the active and standby nodes.
Step 1 Run the SAPHanaSR-showAttr command to check the System Replication active/standby
status.
hw00001:/home # SAPHanaSR-showAttr
Host \ Attr clone_state remoteHost roles site srmode
sync_state vhost lpa_sle_lpt
----------------------------------------------------------------------------------
------------------------------
hw00001 PROMOTED hw00002 4:P:master1:master:worker:master hw00001
sync PRIM hw00001 1416991408
hw00002 DEMOTED hw00001 4:S:master1:master:worker:master hw00002
sync SOK hw00002 30
sync_state has the following types:

l PRIM indicates that the node is the active node.
l SOK indicates that the node is ready and synchronization is complete.
l SFAIL indicates that synchronization is not complete.
----End

The standby node of the cluster takes over the database only when data synchronization
between the standby node and the active node is complete.
If the data is not synchronized or the database on the active node is faulty, the standby node
does not take over the database from the active node.
Step 1 Run the SAPHanaSR-showAttr command to check the active/standby status of the SAP
HANA system replication.
Run the SAPHanaSR-showAttr command on the active node.
The result description is as follows:
sync_state:
PRIM indicates that the node is the active node.
SOK indicates that the node is ready and synchronization is complete.
SFAIL indicates that synchronization is not complete.
The command output is similar to the following:
HW00001:/home # SAPHanaSR-showAttr
Host \ Attr clone_state remoteHost roles site srmode sync_state
vhost lpa_sle_lpt
----------------------------------------------------------------------------------
-----

HW00001 PROMOTED HW00002 4:P:master1:master:worker:master HW00001 sync PRIM

HW00001 1416991408
HW00002 DEMOTED HW00001 4:S:master1:master:worker:master HW00002 sync SOK
HW00002 30



----End
9.11 Recovery Operation and Procedure After HA

Takeover
By default, the automatic registration function is disabled. After an HA takeover is performed,
the original active node needs to be registered with the current active node. You need to
manually run the registration command. For details about the registration procedure, see Step
1 to Step 2. After the registration is complete, clear the failure count of the resource to enable
the resource to run again.
NOTE
The SLES 12 updates the failure counting mechanism. After an HA takeover, the database needs to be
registered and SAP HANA resources are cleaned up again.
Step 1 Assume that the original active node is HW00001 and the original standby node is
HW00002. After HW00002 takeover is complete, virtual IP resources are migrated to
HW00002.
Step 2 Run the hdbnsutil -sr_register --remoteHost=hw00002 --remoteInstance=00 --

replicationMode=sync --name=hw00001 --operationMode=logreplay command using the
hw00001 database account of the original active node to register hw00001 as the standby
node with the hw00002.

Step 3 Log in to SUSE Hawk, choose Cleanup in the Operations column of the SAP HANA
resource record to clear the resource failure count.
Step 4 After the failure count is cleaned up, the database automatically starts on HW00001.
----End

Guide (CH121&CH242&2288H&2488H&9008 V5) 10 Cluster HA Configuration (RHEL)
10 Cluster HA Configuration (RHEL)
10.1 Configuring Basic Functions

10.3 HA Cluster Takeover Prerequisite
10.1 Configuring Basic Functions

Step 1 On the active and standby nodes, run the following commands to use the yum repository to
install the RPM packages:
NOTE
For details about how to configure the yum source mode, see 3 OS and Database Installation.
yum list
yum install -y pacemaker corosync
yum localinstall resource-agents-sap-3.9.5-105.el7_4.2.x86_64.rpm
yum localinstall resource-agents-sap*
yum install pcs fence-agents-all
yum install gtk2 libicu xulrunner sudo tcsh libssh2 expect cairo graphviz iptraf-
ng krb5-workstation krb5-libs libpng12 nfs-utils lm_sensors rsyslog openssl
PackageKit-gtk3-module libcanberra-gtk2 libtool-ltdl xorg-x11-xauth numactl
xfsprogs net-tools bind-utils openssl098e tuned tuned-utils libtool-ltdl ntp
Step 2 Run the HDB -info command on the active and standby nodes to check whether the SAP
HANA database is running properly. If the processes in bold are displayed, the database is
running properly.
Active node:
s00adm@hw00001:/usr/sap/S00/HDB00> HDB -info
USER PID PPID %CPU VSZ RSS COMMAND
s00adm 21117 21116 0.8 116304 2972 -sh
s00adm 21256 21117 0.0 113256 1644 \_ /bin/sh /usr/sap/S00/HDB00/HDB -
info
s00adm 21288 21256 0.0 139504 1644 \_ ps fx -U s00adm -o
user,pid,ppid,pcpu,vsz,rss,args
s00adm 9493 1 0.0 23616 1712 sapstart pf=/usr/sap/S00/SYS/profile/
S00_HDB00_hw00001
s00adm 9520 9493 0.1 349668 33244 \_ /usr/sap/S00/HDB00/hw00001/trace/
hdb.sapS00_HDB00 -d -nw -f /usr/sap/S00/HDB00/hw00001/daemon.ini
pf=/usr/sap/S00/SYS/profile/S00_HDB00_hw00001
s00adm 9542 9520 39.1 8876924 4973812 \_ hdbnameserver
s00adm 9885 9520 29.7 4398064 1449272 \_ hdbcompileserver
s00adm 9887 9520 5.7 4152936 471824 \_ hdbpreprocessor

s00adm 9932 9520 40.1 11704136 7668544 \_ hdbindexserver -port 30003

s00adm 9939 9520 5.3 6426572 1602784 \_ hdbxsengine -port 30007
s00adm 12308 9520 2.7 3646396 460976 \_ hdbwebdispatcher
s00adm 7377 1 0.0 500188 21908 /usr/sap/S00/HDB00/exe/sapstartsrv
pf=/usr/sap/S00/SYS/profile/S00_HDB00_hw00001 -D -u s00adm
Standby node:
s00adm@hw00002:/usr/sap/S00/HDB00> HDB -info
Step 3 On the active node, run the ./hdbsql -u system -i 00 "select value from
"SYS"."M_INIFILE_CONTENTS" where key='log_mode'" command to set log_mode
to normal for SAP HANA.
Go to the /usr/sap/hdbclient directory and run the following command. Enter the database
password when prompted. In this example, the password is Huawei12#$.
[root@hw00001 hdbclient]# ./hdbsql -u system -i 00 "select value from
"SYS"."M_INIFILE_CONTENTS" where key='log_mode'"
Password:
VALUE
"normal"
1 row selected (overall time 133.117 msec; server time 111.782 msec)
Step 4 On the active node, run the grep Autostart /usr/sap/S00/SYS/profile/* command to check
whether the automatic startup function of the SAP HANA database is disabled.
Autostart = 0 indicates that the automatic startup function is disabled.

[root@hw00001 hdbclient]# grep Autostart /usr/sap/S00/SYS/profile/*
/usr/sap/S00/SYS/profile/S00_HDB00_hw00001:Autostart = 0
Step 5 Run the systemctl start pcsd.service and systemctl enable pcsd.service commands on the
active and standby nodes to start the pcs service and enable the automatic startup function.
Active node:
[root@hw00001 hdbclient]# systemctl start pcsd.service
[root@hw00001 hdbclient]# systemctl enable pcsd.service
Standby node:
[root@hw00002 hdbclient]# systemctl start pcsd.service
[root@hw00002 hdbclient]# systemctl enable pcsd.service
Step 6 Run the passwd hacluster command on the active and standby nodes to change the cluster
password.
In this example, hacluster is the cluster name and linux is the password. Change them based
on the actual situation.
Active node:
[root@hw00001 ~]# passwd hacluster
Changing password for user hacluster.
New password: (Enter the password.)
Retype new password: (Confirm the password.)
passwd: all authentication tokens updated successfully.
[root@hw00001 ~]#
Standby node:
[root@hw00002 ~]# passwd hacluster
Changing password for user hacluster.
New password: (Enter the password.)
Retype new password: (Confirm the password.)
passwd: all authentication tokens updated successfully.
[root@hw00002 ~]#
Step 7 Run the pcs cluster auth 10.5.5.131 10.5.5.142 command on the active and standby nodes to
perform authentication.

hacluster is the cluster name, linux is the password (configured at step 6), and 10.5.5.131
and 10.5.5.142 are SR channel network port IP addresses of the active and standby nodes.
Change them based on the actual situation.
[root@hw00002 ~]# pcs cluster auth 10.5.5.131 10.5.5.142
Username: hacluster
Password:
10.5.5.142: Authorized
10.5.5.131: Authorized
Step 8 Run the pcs cluster setup --name hacluster --start 10.5.5.131,192.126.126.131
10.5.5.142,192.126.126.142 --transport udpu command on the active and standby nodes to
initialize clusters and set the cluster communication mode to unicast.
In this example, the OS management network port is used as the cluster communication
network port. Before configuration, ensure that the OS management network port is normal.
10.5.5.131 and 10.5.5.142 are the SR channel network port IP addresses of the two hosts in
the cluster. 192.126.126.131 and 192.126.126.142 are OS management network port IP
addresses of the two hosts in the cluster. Change them based on the actual situation.
Active node:
[root@hw00001 ~]# pcs cluster setup --name hacluster --start
10.5.5.131,192.126.126.131 10.5.5.142,192.126.126.142 --transport udpu
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
hw00001: Succeeded
hw00002: Succeeded
Starting cluster on nodes: hw00001, hw00002...
hw00001: Starting Cluster...
Synchronizing pcsd certificates on nodes hw00001, hw00002...
hw00002: Success
hw00001: Success
Restaring pcsd on the nodes in order to reload the certificates...

hw00002: Success
hw00001: Success
[root@hw00001 ~]#
Standby node:
[root@hw00002 ~]# pcs cluster setup --name hacluster --start
10.5.5.131,192.126.126.131 10.5.5.142,192.126.126.142 --transport udpu --force
Step 9 Run the pcs cluster auth hw00001 hw00002 command on the active and standby nodes to
perform authentication.
In this example, hacluster is the cluster name, linux is the password set in the previous step,
hw00001 is the host name of the active node, and hw00002 is the host name of the standby
node. Change them based on the actual situation.
[root@hw00002 ~]# pcs cluster auth hw00001 hw00002
Username: hacluster
Password:
hw00002: Authorized
hw00001: Authorized
Step 10 Run the pcs cluster setup --name hacluster --start hw00001,hw00001sr
hw00002,hw00002sr --transport udpu command on the active and standby nodes to
initialize the cluster and set the cluster communication mode to UDPU communication.

This example uses the upper-layer service ports as cluster communication ports. Before the
configuration, ensure that the upper-layer service ports of the cluster nodes can communicate
with each other properly.
hw00001 and hw00002 are the names mapping to the upper-layer service ports of the two
cluster nodes. hw00001sr and hw00002sr are the names mapping to the cluster SR channel
ports. Change them based on the actual situation.
Active node:
[root@hw00001 ~]# pcs cluster setup --name hacluster --start hw00001,hw00001sr
hw00002,hw00002sr --transport udpu
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
hw00001: Succeeded
hw00002: Succeeded
Starting cluster on nodes: hw00001, hw00002...
Synchronizing pcsd certificates on nodes hw00001, hw00002...
hw00002: Success
hw00001: Success
Restaring pcsd on the nodes in order to reload the certificates...

hw00002: Success
hw00001: Success
[root@hw00001 ~]#
Standby node:
[root@hw00002 ~]# pcs cluster setup --name hacluster --start hw00001,hw00001sr
hw00002,hw00002sr --transport udpu --force
Step 11 On the active node, run the following commands to start the services related to the cluster:
[root@hw00001 ~]# systemctl start pcsd.service
[root@hw00001 ~]# systemctl start corosync.service
[root@hw00001 ~]# systemctl start pacemaker.service
Step 12 On the active node, run the following commands to enable the automatic startup function for
pacemaker, corosync, and pcsd.
[root@hw00001 ~]# systemctl enable pcsd.service
[root@hw00001 ~]# systemctl enable corosync.service
[root@hw00001 ~]# systemctl enable pacemaker.service
Step 13 On the active node, run the pcs status command to check the two-node cluster status.
[root@hw00001 ~]# pcs status
Cluster name: hacluster
WARNING: no stonith devices and stonith-enabled is not false
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Fri Oct 13 16:53:31 2017Last change: Fri Oct 13 16:45:38 2017 by
hacluster via crmd on hw00001
Stack: corosync
Current DC: hw00001 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
2 nodes and 0 resources configured
Online: [ hw00001 hw00002 ]
PCSD Status:
hw00001: Online
hw00002: Online

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@hw00001 ~]#
Step 14 On the active node, run the corosync-cfgtool -s command to check the heartbeat status.
[root@hw00001 ~]# corosync-cfgtool -s
Local node ID 1
RING ID 0
id= 10.5.1.10
RING ID 1
id= 10.5.2.10
[root@hw00001 ~]#
----End

Step 1 Log in to the iBMC, choose Configuration > Local User, and grant the IPMI LAN login
interface permission to the root user.
Step 2 On the active node, run the following commands to configure the basic two-node cluster
resource parameters.
[root@hw00001 ~]# pcs property set no-quorum-policy="stop"
[root@hw00001 ~]# pcs resource defaults default-resource-stickness=1000
[root@hw00001 ~]# pcs resource defaults default-migration-threshold=5000
[root@hw00001 ~]# pcs resource op defaults timeout=600s
Step 3 On the active node, run the following commands to configure the stonith resource instance to
restart the peer server by IPMI when brain split occurs. (\> is the connector. To copy the text,
remove the symbol and then execute it.)
[root@hw00001 ~]# pcs stonith create st_ipmi_hw00001 fence_ipmilan \
> ipaddr=126.126.126.14 \
> lanplus=on \
> login="Administrator" \
> passwd="Admin@9000" \
> pcmk_host_list="hw00001"
[root@hw00001 ~]# pcs stonith create st_ipmi_hw00002 fence_ipmilan \

> ipaddr=126.126.126.16 \
> lanplus=on \

> login="Administrator" \
> passwd="Admin@9000" \
> pcmk_host_list="hw00002"
st_ipmi_hw00001 Indicates the stonith resource name. Change it based on the

actual host name.
fence_ipmilan Indicates that the port type is ipmilan.
ipaddr Indicates the iBMC IP address of the local node. In this

example, the iBMC IP address of the node hw00001 is
126.126.126.14.
login Indicates the username for logging in to the iBMC. In this

example, the username is Administrator.
passwd Indicates the password for logging in to the iBMC. In this

example, the password is Admin@9000.
pcmk_host_list Indicates the host name that corresponds to the stonith

resource. In this example, the host name that corresponds to
the resource st_ipmi_hw00001 is hw00001.
Step 4 On the active node, run the following commands to configure the stonith resource constraint.
The resource st_ipmi_hw00001 does not run on the node hw00001 and the resource
st_ipmi_hw00002 does not run on the node hw00002
[root@hw00001 ~]# pcs constraint location st_ipmi_hw00001 avoids hw00001
[root@hw00001 ~]# pcs constraint location st_ipmi_hw00002 avoids hw00002
Step 5 On the active node, run the pcs status command to check that the stonith resource is
configured successfully.
[root@hw00001 ~]# pcs status
Last updated: Fri Oct 13 18:12:00 2017Last change: Fri Oct 13 18:11:35 2017 by
root via cibadmin on hw00001
Stack: corosync
Online: [ hw00001 hw00002 ]
st_ipmi_hw00001(stonith:fence_ipmilan):Started hw00002
PCSD Status:
hw00001: Online
hw00002: Online
Daemon Status:
[root@hw00001 ~]#

Step 6 On the active and standby nodes, check whether the cluster iBMC IPMI link is connected
properly.
Run the ipmitool -I lanplus -H 192.168.1.232 -U root -P Huawei12#$ chassis power status
command on the active and standby nodes to check the power status of the peer nodes. If the
command output is "Chassis Power is on", the IPMI link is connected properly.
Active node:
[root@hw00001 ~]# ipmitool -I lanplus -H 126.126.126.16 -U Administrator -P
Admin@9000 chassis power status
Get Device ID command failed: 0xc1 Invalid command
Chassis Power is on
Standby node:
[root@hw00002 ~]# ipmitool -I lanplus -H 126.126.126.14 -U Administrator -P
Admin@9000 chassis power status
Get Device ID command failed: 0xc1 Invalid command
Chassis Power is on
[root@hw00002 ~]#

-H 192.168.1.232 Indicates the iBMC IP address of the peer node. Replace it

with the actual IP address.
-U root Indicates the username for logging in to the iBMC of the peer
node.
-P Huawei12#$ Indicates the password for logging in to the iBMC of the peer
node. Replace it with the actual password.
Step 7 On the active node, run the pcs resource create rsc_ip_SAPHana_S00_HDB00 IPaddr2 \
command to configure the virtual IP address. The virtual IP address and service IP address
must be in the same network segment.
[root@hw00001 ~]# pcs resource create rsc_ip_SAPHana_S00_HDB00 IPaddr2 \
> ip="10.5.1.12" \
> iflabel=0
[root@hw00001 ~]#

rsc_ip_SAPHana_<SID> Indicates the name of the virtual IP address. The SID must be
_HDB00 changed based on the actual situation. In this example, the SID
is S00.
Step 8 On the active node, run the pcs resource create rsc_SAPHanaTopology_S00_HDB00
SAPHanaTopology \ command to create a clone resource.
[root@hw00001 yum.repos.d]# pcs resource create rsc_SAPHanaTopology_S00_HDB00
SAPHanaTopology \
> SID=S00 \
> InstanceNumber=00 \
> op start timeout=600 \

> op stop timeout=300 \

> op monitor interval=10 timeout=600
[root@hw00001 yum.repos.d]#
rsc_SAPHanaTopology_ Indicates the name of the clone resource. The SID must be
<SID>_HDB00 changed based on the actual situation. In this example, the SID
is S00.
InstanceNumber=00 Indicates the ID of the database instance. In this example, the

ID is 00.
Step 9 On the active node, run the pcs resource clone rsc_SAPHanaTopology_S00_HDB00 \
command to configure clone resource parameters.
[root@hw00001 yum.repos.d]# pcs resource clone rsc_SAPHanaTopology_S00_HDB00 \
> meta is-managed=true clone-node-max=1 target-role="Started" interleave=true
rsc_SAPHanaTopology_ Indicates the name of the clone resource. The SID must be
<SID>_HDB00 changed based on the actual situation. In this example, the SID
is S00.
Step 10 On the active node, run the pcs resource create rsc_SAPHana_S00_HDB00 SAPHana \
command to create the SAP HANA active and standby resource.
[root@hw00001 ~]# pcs resource create rsc_SAPHana_S00_HDB00 SAPHana \
> SID=S00 \
> InstanceNumber=00 \
> PREFER_SITE_TAKEOVER=true \
> DUPLICATE_PRIMARY_TIMEOUT=7200 \
> AUTOMATED_REGISTER=false \
> op start timeout=3600 \
> op stop timeout=3600 \
> op promote timeout=3600 \
> op demote timeout=3600 \
> op monitor interval=59 role="Master" timeout=700 \
> op monitor interval=61 role="Slave" timeout=700
[root@hw00001 ~]#
rsc_SAPHana_S00_HDB Indicates the name of the SAP HANA resource. The SID must
00 be changed based on the actual situation. In this example, the
SID is S00.
PREFER_SITE_TAKEO Indicates whether to enable the automatic takeover function. In

VER this example, this function is enabled.

DUPLICATE_PRIMAR Indicates the SR takeover timeout interval.

Y_TIMEOUT
AUTOMATED_REGIST Indicates whether to enable the automatic registration function.

ER In this example, this function is disabled.
Step 11 On the active node, run the pcs resource master msl_rsc_SAPHana_S00_HDB00
rsc_SAPHana_S00_HDB00 \ command to configure the active and standby resource
parameters.
[root@hw00001 hdbclient]# pcs resource master msl_rsc_SAPHana_S00_HDB00
rsc_SAPHana_S00_HDB00 \
> meta is-managed=true notify=true clone-max=2 clone-node-max=1 \
> target-role="Started" interleave=true
[root@hw00001 hdbclient]#
Step 12 On the active node, run the following commands to configure the resource constraint and
resource sequence.
[root@hw00001 ~]# pcs constraint colocation add rsc_ip_SAPHana_S00_HDB00 with
master msl_rsc_SAPHana_S00_HDB00 2000
[root@hw00001 ~]# pcs constraint order rsc_SAPHanaTopology_S00_HDB00-clone then
msl_rsc_SAPHana_S00_HDB00 symmetrical=false
Adding rsc_SAPHanaTopology_S00_HDB00-clone msl_rsc_SAPHana_S00_HDB00 (kind:
Mandatory) (Options: first-action=start then-action=start symmetrical=false)
[root@hw00001 ~]#
Step 13 On the active node, run the pcs resource create bond_vip-monitor ethmonitor
interface=bond_vip --clone command to create a clone resource for the upstream service
port.
In this example, bond_vip is the upstream service port. Change it based on the actual
situation.
[root@hw00001 ~]# pcs resource create bond_vip-monitor ethmonitor
interface=bond_vip --clone
Step 14 On the active node, run the pcs constraint location rsc_ip_SAPHana_S00_HDB00 rule
score=-INFINITY ethmonitor-bond_vip ne 1 command to configure resource constraint to
forbid the virtual IP address to run on the node when the upstream service port is faulty.
In this example, rsc_ip_SAPHana_S00_HDB00r is the name of the virtual IP address and
bond_vip is the upstream service port. Change them based on the actual situation.
[root@hw00001 ~]# pcs constraint location rsc_ip_SAPHana_S00_HDB00 rule score=-
INFINITY ethmonitor-bond_vip ne 1
Step 15 On the active node, run the following commands to enable the automatic startup function for
pacemaker, corosync, and pcsd.
[root@hw00001 ~]# systemctl enable pcsd.service
[root@hw00001 ~]# systemctl enable corosync.service
[root@hw00001 ~]# systemctl enable pacemaker.service
Step 16 On the active node, run the pcs resource cleanup command to obtain the latest resource
status.
[root@hw00001 ~]# pcs resource cleanup
Step 17 On the active node, run the pcs status command to check the two-node cluster status.

NOTE
The command output of pcs resource cleanup will be displayed in 5 minutes. Check the two-node
cluster status after the latest resource status is obtained.
[root@hw00001 yum.repos.d]# pcs status
Last updated: Mon Oct 16 11:19:36 2017Last change: Mon Oct 16 11:19:21 2017 by
root via cibadmin on hw00001
Stack: corosync
Online: [ hw00001 hw00002 ]
rsc_ip_SAPHana_S00_HDB00(ocf::heartbeat:IPaddr2):Started hw00001
Master/Slave Set: msl_rsc_SAPHana_S00_HDB00 [rsc_SAPHana_S00_HDB00]
Slaves: [ hw00002 ]
Clone Set: bond_vip-monitor-clone [bond_vip-monitor]
Started: [ hw00001 hw00002 ]
Clone Set: rsc_SAPHanaTopology_S00_HDB00-clone [rsc_SAPHanaTopology_S00_HDB00]
Started: [ hw00001 hw00002 ]
PCSD Status:
hw00001: Online
hw00002: Online
Daemon Status:
----End
10.3 HA Cluster Takeover Prerequisite

The prerequisite for automatic HA cluster takeover is that data synchronization is complete.
The system replication duration required by the HANA database depends on the data volume
of the HANA database and the network bandwidth of the system synchronization channel.
Step 1 Log in to the active node as the root user, and run the su - s00adm command to switch to the
HANA database user.
NOTE
Replace s00 in the command with the actual database SID in lowercase.
Step 2 Run the cdpy command as a database user to go to the python directory.
Step 3 On the active node, run the python systemReplicationStatus.py command using a database
account to check the data synchronization status.
If the displayed status is ACTIVE, the database has completed synchronization and remains
in sync mode, and you can perform the database takeover operation.
If the status is Initializing, the database is synchronizing data and is not ready for the
takeover operation.
The following is an example of the command output.

s00adm@hw00001:/usr/sap/POC/HDB00> cdpy
s00adm@hw00001:/usr/sap/S00/HDB00/exe/python_support> python
systemReplicationStatus.py
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name |
Secondary | Secondary | Secondary | Secondary | Secondary | Replication |
Replication | Replication |
| | | | | | | |
Host | Port | Site ID | Site Name | Active Status | Mode |
Status | Status Details |
| -------- | ---- | ----- | ------------ | --------- | ------- | --------- |
--------- | --------- | --------- | --------- | ------------- | ----------- |
----------- | -------------- |
| SYSTEMDB | hw00001 | 30001 | nameserver | 1 | 1 | hw00001
| hw00002 | 30001 | 2 | hw00002 | YES |
SYNC | ACTIVE | |
| TD1 | hw00001 | 30040 | indexserver | 2 | 1 | hw00001
| hw00002 | 30040 | 2 | hw00002 | YES |
SYNC | ACTIVE | |
status system replication site "2": ACTIVE

overall system replication status: ACTIVE
Local System Replication State

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mode: PRIMARY
site id: 1
site name: hw00001
s00adm@hw00001:/usr/sap/S00/HDB00/exe/python_support>
----End

Guide (CH121&CH242&2288H&2488H&9008 V5) 11 OS Lifecycle
11 OS Lifecycle
11.1 SLES for SAP Lifecycle

11.2 RHEL for SAP Lifecycle
11.1 SLES for SAP Lifecycle

SLES for SAP 11 Lifecycle
Figure 11-1 SLES for SAP 11 Lifecycle Model
13-year lifecycle
l 10 years general support

l 3 years extended support
l ~18mon overlay between SPs (ESPOS)
l Up to 4.5 years service pack support
Long Term Service Pack Support (LTSS)
l Up to 2 years for each SP, including GA version

l Up to 3 years for the major version
SLES for SAP 11 Lifecycle Explained
l For the major version:

A total of 13-year support period is provided, 10 years of general & ESPOS support, and
3 years of LTSS.
l For each of the SPs:
– Generally the SPs are released in a around 12-month cadence. When the new SP is
released, the last SP will be continue supported for about 18 months, provide
enough time for the customer to test and migrate to the new SP.
– Each SP, except the last one, has around 18 months of general support and 12
months of Extended SP Overlay Support (ESPOS) period. After that, 2 years of
LTSS support can be provide by SUSE if the customer purchases the LTSS support
in addition to their subscription.
– The last SP will receive longer general & ESPOS support than previous SPs, till the
end of the 10th year since the release of the major version.
NOTE
For detail information about the SLES for SAP lifecycle, visit:
https://scc.suse.com/docs/userguide
SLES for SAP 12 and 15 Lifecycle
Figure 11-2 SLES for SAP 12 and 15 Lifecycle Model
13-year lifecycle
l 10 years general support

l 3 years extended support
l ~3.5-year overlay between SPs
l Up to 4.5 years service pack support
Long Term Service Pack Support (LTSS)
l 3 years for the last SP
SLES for SAP Lifecycle Explained
l For the major version:

A total of 13-year support period is provided, 10 years of general & ESPOS support, and
3 years of LTSS.
l For each of the SPs:

– Generally the SPs are released in a around 12-month cadence. When the new SP is
released, the last SP will be continue supported for about 3.5 years, provide enough
time for the customer to test and migrate to the new SP.
– Each SP, except the last one, has around 18 months of general support and 42
months of Extended SP Overlay Support (ESPOS) period. The ESPOS included in
SLES for SAP subscription now.
– The last SP will receive longer general support than previous SPs, till the end of the
10th year since the release of the major version. There will be a 3-year LTSS
support for the last SP.
– Release plan of SP5 & 6, including whether there will be such SP and when, are
subject to change according to the actual situation around the time of SP3/4.
NOTE
For detail information about the SLES for SAP lifecycle, visit:
https://scc.suse.com/docs/userguide
11.2 RHEL for SAP Lifecycle

Figure 11-3 RHEL for SAP Lifecycle Model
Update Services for SAP Solutions (E4S)

l Red Hat Enterprise Linux for SAP Solutions inherit the policy from EUS and should
apply to Red Hat Enterprise Linux 7.2 and higher.
l Offered as part of RHEL for SAP Solutions bundle only.
– Not offered as separate Add-On.
l Support for 4 years from GA (with the exception of RHEL 7.5).
l Extension of EUS support (Critical Impact Security fixes and selected Urgent Priority
Bug fixes).
l Included in both Premium and Standard subscriptions.
l Support both X86_64 server and Power LE.
l Provided on specific releases only with one new release added annually.
– RHEL 7.2 (through November 30, 2019) - x86 only
– RHEL 7.3 (through November 30, 2020) - x86 and Power LE (no HA)
– RHEL 7.4 (through August 31, 2021) - x86 and Power LE, with HA on both

– RHEL 7.6 (through October 31, 2022)

NOTE
For detail information about the RHEL for SAP lifecycle, visit:
https://access.redhat.com/support/policy/updates/errata/

Guide (CH121&CH242&2288H&2488H&9008 V5) 12 Responsibility Matrix and Problem Handling Process
12 Responsibility Matrix and Problem

Handling Process
12.1 Responsibility Matrix

12.2 Problem Handling Process
12.1 Responsibility Matrix

SAP defines a clear responsibility matrix for SAP HANA projects. The link to the official
website is as follows:
https://help.sap.com/doc/eb3777d5495d46c5b2fa773206bbfb46/2.0.00/en-US/
e243909cbb571014a135a7faa61d61f4.html
12.2 Problem Handling Process

If a problem occurs, contact SAP. SAP will work with Huawei and SUSE to locate the
problem.
l If a SLES OS problem occurs, SAP will forward the problem to the SUSE.

l If an SAP HANA problem occurs, SAP will handle the problem occurs.
l If a hardware or RHEL OS problem occurs, SAP will forward the problem to Huawei for
analysis and handling. If the problem is an evident hardware or RHEL OS problem, you
can also submit a trouble ticket through the Huawei 400 hotline. The Huawei R&D
maintenance will handle the problem.
Figure 12-1 Problem handling process
12.2.1 Vendor Contact Information

SAP
Mainland China: 400-620-2008
Support website: https://support.sap.com
Huawei
Mainland China: 400-822-9999 (enterprise); 400-830-2118 (carrier)
Support website: https://support.huawei.com
SUSE
Mainland China: 400-810-6500
Support website: https://scc.suse.com/
Red Hat
Mainland China: 800 810 2100; 400 890 2100
Support website: https://access.redhat.com
12.2.2 SAP Ticket Submission Process

Step 1 Enter support.sap.com/message in the address box of the browser to access the SAP support
website.

Step 2 On the home page, click the icon in the upper right corner to log in to the SAP. Enter the
subscription account and password to log in to the website.
Step 3 Click Report an incident. The page for generating a trouble ticket is displayed.
Step 4 Enter any character string in Enter search term and press Enter. Click Contact SAP
Support at the lower right corner.

Step 5 Select a product, for example, SAP ERP, and click Search.
Step 6 Go to the problem description page, fill in the trouble ticket details, and click Submit at the
lower right corner.
l Language: English is preferred.
l Priority: Set this parameter based on the impact on customer services. If an incident has
affected customer services, set the parameter to High or Very high. For a root cause
analysis incident, set the parameter to Medium.
l Subject: Briefly describe the symptom, such as database do not respond or system hung.
l Component: If the incident is related to the operating system, select BC-OP-LNX. After
filling in the details, click Submit at the lower right corner.
l Description: Describe the symptom in detail.

----End
12.2.3 Huawei Ticket Submission Process

If the fault persists after taking the above measures, contact Huawei technical support as
follows:
l Contact the Huawei service hotline. For details about the contact information, visit http://
e.huawei.com/en/service-hotline.
For enterprise users in China, contact Huawei in either of the following ways:
– Hotline: 400-822-9999
– Customer service email: support_e@huawei.com
Enterprise customers: Global Service Hotline
For carrier users in China, contact us in the following ways:
– Hotline: 400-830-2118
– Email: support@huawei.com
Enterprise customers: Global Service Hotline
l Contact the technical support personnel of your local Huawei office.
12.2.4 SUSE Ticket Submission Process

Step 1 Use the activated SUSE account to log in to the scc.suse.com. For details about how to
activate the subscription, see the Huawei SAP HANA Appliance Single Node and Two
Node HA Maintenance Guide. After the login, click Support in the MYTOOLS area on the
left of the main page, as shown in Figure 12-2.

Figure 12-2 SUSE support
Step 2 On the Support Tickets page, click Open a new ticket, as shown in Figure 12-3.
Figure 12-3 Open a new ticket
Step 3 On the New Support Ticket page, fill in the ticket and submit it.
1. Set Account, Entitlement, and Product, and click Next.

Figure 12-4 New Support Ticket - PRODUCT
2. Set Ticket Title and click Next.
Figure 12-5 New Support Ticket - TICKET TITLE
3. Select Platform and Severity, describe the problem in the Description text box, and
click Next.

Figure 12-6 New Support Ticket - DESCRIPTION
4. Select a contact method and click Create Support Ticket to create the ticket.
Figure 12-7 New Support Ticket - CONTACT
----End
12.2.5 Red Hat Ticket Submission Process

You can obtain technical support from Red Hat in three ways, support case submission, phone
support, and online service. The following describes the methods.
(Preferred) Online Support Case Submission

l Create a support case.
Online support case submission allows you to share technical data, error information,
and system information with Red Hat technical support engineers. The procedure is as
follows:

a. Log in to the Red Hat Customer Portal and go to the page for creating a support
case: There are two ways to go to the page.
i. Click Support Cases on the white bar at the top of the page, and then click
Open a New Support Case.
ii. Click Open a Support Case on the Red Hat Customer Portal.
b. Select the product involved.
c. Select a proper version from the list.
d. Enter the short description of the problem to be resolved in Case Name.
After you enter the problem description, the recommended solutions are displayed
on the right. You can check whether the solutions can resolve your problem.
e. Enter the details of the problem in Case Description. This helps the Red Hat
engineer to resolve the problem in a timely manner.
f. You are advised to provide logs or other diagnostic files to help support engineers
quickly resolve your problems. You can click Attach Files under Case Description
to attach a file.
g. Select a support level, which depends on the service level you purchase.
h. Select a proper severity for the problem. For details, see the severity definition.
i. (Optional) Specify a user who has a Red Hat login account to receive the email
notification about the support case.
j. (Optional) Select the support case group for the support case.
k. Click Submit at the bottom.
Fill in all the required fields before submitting the support case. If the Submit
button is unavailable, check whether all required fields are set. After this form is
submitted, your support case will be recorded. The case details page is displayed.

l Viewing and Updating a Ticket

After submitting the support case, it is recommended that you and Red Hat engineers
communicate with each other to quickly resolve the problem. You can view historical

records and add updates in the Case Discussion section. To add an update, enter the
content in the text field and click Post.
At the same time, after you submit the support case, you will receive the update of the
support case by email. You can directly reply to the email (you are advised to remove the
original email content) to update the support case and interact with the Red Hat support
engineers. To add attachment, log in to the Red Hat Customer Portal and submit the
information on the Case page.
If the problem has been resolved, you can close the support case. After the support case
is closed, you may receive a satisfaction survey email from Red Hat. Please participate in
the survey and provide your valuable comments and suggestions. This will help Red Hat
better serve you later.
If the problem persists after the case is closed, you can find the case and update the reply.
Then you can enable the case again.
For details about how to create and manage support cases, see the following document:
How do I open and manage a support case on the Customer Portal?
Vendor Contact Information

To better share technical data, error information, and system information, you are advised to
create a support case before calling.
Red Hat Technical Support Hotline
Mainland China:

l Fixed-line phone: 800 810 2100 (option 2)

l Mobile phone: 400 890 2100 (option 2)
Hong Kong: 800 901 222
Taiwan: 00801 861036
Support website: https://access.redhat.com
Online Service
Red Hat offers online services. You can click Support Cases or Open a New Support Case,
and click the chat support button in the upper right corner of the support case list or creation
page to initiate an online service session, as shown in the following figure.
For details about Red Hat online support, see the information at the following addresses:
Red Hat Chat Support: https://access.redhat.com/articles/313583
Red Hat Technical Support reference guide:
https://access.redhat.com/sites/default/files/attachments/chinese-
reference_guide_to_engaging_with_red_hat_support_brochure.pdf
Support Case Escalation

If you feel your support case requires extra attention, do as follows to escalate it:
1. From the Support Case List, select the case you wish to escalate and click Request
Management Escalation on the right under recommended solutions.

2. In the dialog box, be as specific as you can regarding your area of concern, business
impact and expectations. Click Submit.
A support manager will contact you within 4 hours. Your Red Hat sales representative or
technical account manager (if applicable) can also escalate on your behalf.

Guide (CH121&CH242&2288H&2488H&9008 V5) 13 Support and Other Resources
13 Support and Other Resources
13.1 Technical Support
13.1 Technical Support

Huawei's timely and efficient response is available from:
l Local branch offices

l Secondary technical support system
l Telephone support
l Remote support
l Onsite support
Technical Support Website

Technical documents are available on the technical support website: http://e.huawei.com/
Self-Service Platform and Community

Learn more about servers and communicate with experts at:
l Huawei Server Information Self-Service Platform for specific server product

documentation.
l Huawei server intelligent Q&A system for quick learning about products.
l Huawei Enterprise Support Community (Server) for learning and discussion.
News
For notices about product life cycles, warnings, and updates, visit Product Bulletins.
Cases
Learn about server applications at Knowledge Base.

Guide (CH121&CH242&2288H&2488H&9008 V5) 13 Support and Other Resources
Huawei Technical Support

If a fault persists after taking the above measures, contact technical support at your local
Huawei office. If a local Huawei office is not available, contact Huawei technical support as
follows:
l Enterprise customers:
Send emails to support_e@huawei.com or visit Global Service Hotline.
l Carriers
Send emails to support@huawei.com or visit Global TAC Information.

Guide (CH121&CH242&2288H&2488H&9008 V5) A Appendix 1
A Appendix 1
The following describes the internal networking of compute nodes.
Step 1 Open the browser, enter https://192.168.34.112 in the address box, and log in to the MM910
WebUI.
Step 2 Select a compute node and click Network Card Physical Connection Diagram.
Step 3 Click Generate.

Step 4 Select the network ports of two MZ312 cards to view the connections between the NICs and
switch modules. Select different NICs and bind them to different switch modules to improve
network redundancy.
Step 5 Check the serial number of the network port of the OS based on the MAC address of an NIC.

----End

Huawei SAP HANA Appliance Two Node Installation B Software Version List of Red Hat 7.4 Patch Packages in
Guide (CH121&CH242&2288H&2488H&9008 V5) OneBox
B Software Version List of Red Hat 7.4 Patch

Packages in OneBox
audit-2.8.1-3.el7.x86_64.rpm
audit-libs-2.8.1-3.el7.x86_64.rpm
audit-libs-python-2.8.1-3.el7.x86_64.rpm
autogen-libopts-5.18-5.el7.x86_64.rpm
checkpolicy-2.5-6.el7.x86_64.rpm
clufter-bin-0.77.0-2.el7.x86_64.rpm
clufter-common-0.77.0-2.el7.noarch.rpm
compat-sap-c++-5-5.3.1-10.el7_3.x86_64.rpm
compat-sap-c++-6-6.3.1-1.el7_3.x86_64.rpm
corosync-2.4.3-2.el7_5.1.x86_64.rpm
corosynclib-2.4.3-2.el7_5.1.x86_64.rpm
cpp-4.8.5-28.el7_5.1.x86_64.rpm
dbus-1.10.24-7.el7.x86_64.rpm
dbus-libs-1.10.24-7.el7.x86_64.rpm
dwz-0.11-3.el7.x86_64.rpm
expect-5.45-14.el7_1.x86_64.rpm
fence-agents-all-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-amt-ws-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-apc-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-apc-snmp-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-bladecenter-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-brocade-4.0.11-86.el7_5.2.x86_64.rpm

fence-agents-cisco-mds-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-cisco-ucs-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-common-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-compute-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-drac5-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-eaton-snmp-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-emerson-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-eps-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-heuristics-ping-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-hpblade-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-ibmblade-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-ifmib-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-ilo2-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-ilo-moonshot-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-ilo-mp-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-ilo-ssh-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-intelmodular-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-ipdu-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-ipmilan-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-kdump-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-mpath-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-rhevm-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-rsa-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-rsb-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-sbd-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-scsi-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-vmware-rest-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-vmware-soap-4.0.11-86.el7_5.2.x86_64.rpm
fence-agents-wti-4.0.11-86.el7_5.2.x86_64.rpm
fence-virt-0.3.2-13.el7.x86_64.rpm
flac-libs-1.3.0-5.el7_1.x86_64.rpm
gcc-4.8.5-28.el7_5.1.x86_64.rpm
gd-2.0.35-26.el7.x86_64.rpm

ghostscript-9.07-28.el7_4.2.x86_64.rpm
ghostscript-fonts-5.50-32.el7.noarch.rpm
glibc-2.17-222.el7.x86_64.rpm
glibc-common-2.17-222.el7.x86_64.rpm
glibc-devel-2.17-222.el7.x86_64.rpm
glibc-headers-2.17-222.el7.x86_64.rpm
gnutls-dane-3.3.26-9.el7.x86_64.rpm
gnutls-utils-3.3.26-9.el7.x86_64.rpm
graphviz-2.30.1-21.el7.x86_64.rpm
gsm-1.0.13-11.el7.x86_64.rpm
gstreamer1-1.10.4-2.el7.x86_64.rpm
ipmitool-1.8.18-7.el7.x86_64.rpm
iptraf-ng-1.1.4-6.el7.x86_64.rpm
kernel-3.10.0-862.2.3.el7.x86_64.rpm
kernel-devel-3.10.0-862.2.3.el7.x86_64.rpm
kmod-kvdo-6.1.0.168-16.el7_5.x86_64.rpm
krb5-libs-1.15.1-19.el7.x86_64.rpm
krb5-workstation-1.15.1-19.el7.x86_64.rpm
libasyncns-0.8-7.el7.x86_64.rpm
libcanberra-0.30-5.el7.x86_64.rpm
libcanberra-gtk2-0.30-5.el7.x86_64.rpm
libcanberra-gtk3-0.30-5.el7.x86_64.rpm
liberation-fonts-common-1.07.2-16.el7.noarch.rpm
liberation-sans-fonts-1.07.2-16.el7.noarch.rpm
libfontenc-1.1.3-3.el7.x86_64.rpm
libgcc-4.8.5-28.el7_5.1.x86_64.rpm
libgomp-4.8.5-28.el7_5.1.x86_64.rpm
libICE-1.0.9-9.el7.x86_64.rpm
libicu-50.1.2-15.el7.x86_64.rpm
libkadm5-1.15.1-19.el7.x86_64.rpm
libmpc-1.0.1-3.el7.x86_64.rpm
libqb-1.0.1-6.el7.x86_64.rpm
libselinux-2.5-12.el7.x86_64.rpm

libselinux-python-2.5-12.el7.x86_64.rpm
libselinux-utils-2.5-12.el7.x86_64.rpm
libsemanage-2.5-11.el7.x86_64.rpm
libsemanage-python-2.5-11.el7.x86_64.rpm
libsepol-2.5-8.1.el7.x86_64.rpm
libSM-1.2.2-2.el7.x86_64.rpm
libsndfile-1.0.25-10.el7.x86_64.rpm
libstdc++-4.8.5-28.el7_5.1.x86_64.rpm
libstdc++-devel-4.8.5-28.el7_5.1.x86_64.rpm
libvpx-1.3.0-5.el7_0.x86_64.rpm
libwsman1-2.6.3-3.git4391e5c.el7.x86_64.rpm
libXaw-1.0.13-4.el7.x86_64.rpm
libXfont-1.5.2-1.el7.x86_64.rpm
libXmu-1.1.2-2.el7.x86_64.rpm
libXpm-3.5.12-1.el7.x86_64.rpm
libXt-1.1.5-3.el7.x86_64.rpm
libyaml-0.1.4-11.el7_0.x86_64.rpm
linux-firmware-20180220-62.git6d51311.el7.noarch.rpm
lm_sensors-3.4.0-4.20160601gitf9185e5.el7.x86_64.rpm
mozilla-filesystem-1.9-11.el7.x86_64.rpm
mpfr-3.1.1-4.el7.x86_64.rpm
net-snmp-libs-5.7.2-33.el7_5.2.x86_64.rpm
net-snmp-utils-5.7.2-33.el7_5.2.x86_64.rpm
nfs-utils-1.3.0-0.54.el7.x86_64.rpm
ntp-4.2.6p5-28.el7.x86_64.rpm
ntpdate-4.2.6p5-28.el7.x86_64.rpm
numactl-2.0.9-7.el7.x86_64.rpm
OpenIPMI-modalias-2.0.23-2.el7.x86_64.rpm
openssl-1.0.2k-12.el7.x86_64.rpm
openssl-libs-1.0.2k-12.el7.x86_64.rpm
openwsman-python-2.6.3-3.git4391e5c.el7.x86_64.rpm
overpass-fonts-2.1-1.el7.noarch.rpm
pacemaker-1.1.18-11.el7_5.2.x86_64.rpm

pacemaker-cli-1.1.18-11.el7_5.2.x86_64.rpm
pacemaker-cluster-libs-1.1.18-11.el7_5.2.x86_64.rpm
pacemaker-libs-1.1.18-11.el7_5.2.x86_64.rpm
PackageKit-glib-1.1.5-2.el7_5.x86_64.rpm
PackageKit-gtk3-module-1.1.5-2.el7_5.x86_64.rpm
patch-2.7.1-10.el7_5.x86_64.rpm
pcs-0.9.162-5.el7_5.1.x86_64.rpm
perl-srpm-macros-1-8.el7.noarch.rpm
perl-Thread-Queue-3.02-2.el7.noarch.rpm
perl-TimeDate-2.30-2.el7.noarch.rpm
pexpect-2.3-11.el7.noarch.rpm
policycoreutils-2.5-22.el7.x86_64.rpm
policycoreutils-python-2.5-22.el7.x86_64.rpm
poppler-data-0.4.6-3.el7.noarch.rpm
pulseaudio-libs-10.0-5.el7.x86_64.rpm
python-clufter-0.77.0-2.el7.noarch.rpm
python-inotify-0.9.4-4.el7.noarch.rpm
python-IPy-0.75-6.el7.noarch.rpm
python-suds-0.4.1-5.el7.noarch.rpm
PyYAML-3.10-11.el7.x86_64.rpm
redhat-rpm-config-9.1.0-80.el7.noarch.rpm
resource-agents-3.9.5-124.el7.x86_64.rpm
resource-agents-sap-hana-3.9.5-124.el7.x86_64.rpm
rpm-4.11.3-32.el7.x86_64.rpm
rpm-build-4.11.3-32.el7.x86_64.rpm
rpm-build-libs-4.11.3-32.el7.x86_64.rpm
rpm-libs-4.11.3-32.el7.x86_64.rpm
rpm-python-4.11.3-32.el7.x86_64.rpm
rsyslog-8.24.0-16.el7_5.4.x86_64.rpm
ruby-2.0.0.648-33.el7_4.x86_64.rpm
rubygem-bigdecimal-1.2.0-33.el7_4.x86_64.rpm
rubygem-io-console-0.4.2-33.el7_4.x86_64.rpm
rubygem-json-1.7.7-33.el7_4.x86_64.rpm

rubygem-psych-2.0.0-33.el7_4.x86_64.rpm
rubygem-rdoc-4.0.0-33.el7_4.noarch.rpm
rubygems-2.0.14.1-33.el7_4.noarch.rpm
ruby-irb-2.0.0.648-33.el7_4.noarch.rpm
ruby-libs-2.0.0.648-33.el7_4.x86_64.rpm
setools-libs-3.3.8-2.el7.x86_64.rpm
sg3_utils-1.37-12.el7.x86_64.rpm
sound-theme-freedesktop-0.8-3.el7.noarch.rpm
startup-notification-0.12-8.el7.x86_64.rpm
subscription-manager-1.20.11-1.el7_5.x86_64.rpm
subscription-manager-plugin-container-1.20.11-1.el7_5.x86_64.rpm
subscription-manager-rhsm-1.20.11-1.el7_5.x86_64.rpm
subscription-manager-rhsm-certificates-1.20.11-1.el7_5.x86_64.rpm
sudo-1.8.19p2-13.el7.x86_64.rpm
tcl-8.5.13-8.el7.x86_64.rpm
telnet-0.17-64.el7.x86_64.rpm
tuned-2.9.0-1.el7.noarch.rpm
tuned-profiles-sap-hana-2.9.0-1.el7.noarch.rpm
unbound-libs-1.6.6-1.el7.x86_64.rpm
urw-fonts-2.4-16.el7.noarch.rpm
vdo-6.1.0.168-18.x86_64.rpm
xcb-util-0.4.0-2.el7.x86_64.rpm
xorg-x11-font-utils-7.5-20.el7.x86_64.rpm
xorg-x11-xauth-1.0.9-1.el7.x86_64.rpm
xulrunner-31.6.0-2.el7_1.x86_64.rpm
yum-utils-1.1.31-45.el7.noarch.rpm
zlib-devel-1.2.7-17.el7.x86_64.rpm

Huawei SAP HANA Appliance Two Node Installation Guide (CH121&amp;CH242&amp;2288H&amp;2488H&amp;9008 V5) 08

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Huawei SAP HANA Appliance Two Node Installation Guide (CH121&amp;CH242&amp;2288H&amp;2488H&amp;9008 V5) 08

Încărcat de

Drepturi de autor:

Formate disponibile

Huawei SAP HANA Appliance Two

Node Installation Guide

HUAWEI TECHNOLOGIES CO., LTD.

Trademarks and Permissions

Huawei Technologies Co., Ltd.

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. i

About This Document

OSs include the following:

l SUSE Linux Enterprise Server (SLES) 12 SP2/SP3/SP4, 15

SAP HANA databases include the following:

l SAP HANA Database 2.0

l Technical support engineers

Indicates an imminently hazardous situation which, if not

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. ii

Indicates a potentially hazardous situation which, if not

Indicates a potentially hazardous situation which, if not

Indicates a potentially hazardous situation which, if not

Calls attention to important information, best practices

l Added the content of SLES 12 SP4, and SLES 15.

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. iii

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. iv

About This Document.....................................................................................................................ii

3 OS and Database Installation................................................................................................... 17

5 NTP Service Configuration....................................................................................................... 32

6 CHRONY Service Configuration (SLES 15)........................................................................... 35

7 System Replication Configuration...........................................................................................37

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. v

7.7 Checking the System Replication Status...................................................................................................................... 46

8 Cluster HA Configuration (SLES 12)....................................................................................... 48

9 Cluster HA Configuration (SLES 15)....................................................................................... 73

10 Cluster HA Configuration (RHEL).........................................................................................98

12 Responsibility Matrix and Problem Handling Process................................................... 113

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. vi

13 Support and Other Resources............................................................................................... 126

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. vii

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. 1

Figure 1-2 SAP HANA two-node cluster networking (2288H&2488H V5)

Figure 1-3 SAP HANA two-node cluster networking (9008 V5)

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. 2

1. The two 10GE switches need to be stacked to form an HA network.

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. 3

2.1 Hardware Planning

2.1 Hardware Planning

SAP HANA appliance Two SAP HANA Provide computing resources.

Server General-purpose server Functions as an NTP server to

Switch Two 10GE switch Connect the server to the uplink

FC switch Two 8 GB FC switch Connect the server to a storage

Cable l Power cable Connect physical devices in the

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. 4

Table 2-2 Hardware planning (2288H&2488H V5)

SAP HANA appliance Two SAP HANA Provide computing resources.

Server General-purpose server Functions as an NTP server to

Switch Two 10GE switch Connect the server to the uplink

FC switch Two 8 GB FC switch Connect the server to a storage

Cable l Power cable Connect physical devices in the

Table 2-3 Hardware planning (9008 V5)

SAP HANA appliance Two SAP HANA Provide computing resources.

Server General-purpose server Functions as an NTP server to

Switch Two 10GE switch Connect the server to the uplink

FC switch Two 8 GB FC switch Connect the server to a storage

Cable l Power cable Connect physical devices in the

Issue 08 (2019-05-10) Copyright © Huawei Technologies Co., Ltd. 5

2.2 Network Planning

MM910 192.168.34.112 Used to log in to the E9000 management

Huawei SAP HANA Appliance Two Node Installation Guide (CH121&CH242&2288H&2488H&9008 V5) 08

Huawei SAP HANA Appliance Two Node Installation Guide (CH121&CH242&2288H&2488H&9008 V5) 08