Ibm Ds8000

Front cover
IBM System Storage DS8000

Series: Copy Services in
Open Environments
Configuration of Copy Services in
heterogeneous environments
Metro/Global Mirror with

Incremental Resync
TPC for Replication support and

Copy Services with System i
Bert Dufrasne Wenzel Kalabzal

Gustavo Castets Peter Klee
Stephen Baird Markus Oscheka
Werner Bauer Ying Thia
Denise Brown Robert Tondini
Jana Jamsek
ibm.com/redbooks
International Technical Support Organization
IBM System Storage DS8000 Series: Copy Services in

Open Environments
November 2006
SG24-6788-02
Note: Before using this information and the product it supports, read the information in “Notices” on
page xxv.
Third Edition (November 2006)
This edition applies to features, microcode, GUI, and DS CLI as announced for the DS8000 in August 2006.
© Copyright International Business Machines Corporation 2005, 2006. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Contents
Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvi
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvii
The team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvii
Special thanks to . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxix
Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxx
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxx
Summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxxi

November 2006, Third Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxxi
Part 1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Chapter 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Introduction to Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.1 Point-in-Time Copy (FlashCopy). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.2 Remote Mirror and Copy RMC (formerly Peer-to-Peer Remote Copy). . . . . . . . . . 5
Chapter 2. Copy Services architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1 Introduction to the Coppy Services structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.1 Management console defined . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.2 Storage Unit defined . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.3 Storage Facility Image (SFI) defined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.4 What is a Storage Complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 How the new structure of Copy Services works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.1 Remote Mirror and Copy between Storage Complexes . . . . . . . . . . . . . . . . . . . . 13
2.2.2 Differences between the DS CLI and the DS GUI . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 System z communication path for Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Part 2. Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Chapter 3. DS Storage Manager (GUI). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.1 Connecting to the DS8000 DS GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.1.1 Advantages of using the DS GUI over the DS CLI . . . . . . . . . . . . . . . . . . . . . . . . 18
3.1.2 Disadvantages of using the DS GUI over the DS CLI. . . . . . . . . . . . . . . . . . . . . . 19
3.2 Managing the Storage Complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.2.1 Procedure to add a Storage Complex. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3 Creating a DS Storage Management Console PC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.4 Accessing the Information Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Chapter 4. DS Command-Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.1 Introduction and functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.2 Supported operating systems for the DS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3 User accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4 DS CLI profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.5 Command structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.6 Copy Services commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
© Copyright IBM Corp. 2005, 2006. All rights reserved. iii

4.7 Using the DS CLI application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.7.1 Single-shot mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.7.2 Script command mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.7.3 Interactive mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.8 Return codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.9 User assistance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.10 Usage examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Part 3. FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Chapter 5. FlashCopy overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.1 FlashCopy operational environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.3 Basic concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.3.1 Full volume copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.3.2 Nocopy option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.4 FlashCopy in combination with other Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.4.1 FlashCopy and Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.4.2 FlashCopy and Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.4.3 FlashCopy and Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.5 FlashCopy in a DS8300 storage LPAR environment . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Chapter 6. FlashCopy options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.1 Multiple Relationship FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.2 Consistency Group FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.3 FlashCopy target as a Metro Mirror or Global Copy primary. . . . . . . . . . . . . . . . . . . . . 49
6.4 Incremental FlashCopy — refresh target volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.5 Remote FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.6 Persistent FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.7 Reverse restore. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6.8 Fast reverse restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6.9 Options and interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Chapter 7. FlashCopy ordering and activation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

7.1 Ordering FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7.2 Activating FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7.2.1 Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7.2.2 Activation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Chapter 8. FlashCopy interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

8.1 FlashCopy management interfaces - overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.2 DS CLI and DS SM - commands and options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.2.1 Local FlashCopy management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.2.2 Remote FlashCopy management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.3 Local FlashCopy using the DS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
8.3.1 Parameters used with local FlashCopy commands . . . . . . . . . . . . . . . . . . . . . . . 67
8.3.2 Local FlashCopy commands - examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
8.3.3 FlashCopy Consistency Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
8.4 Remote FlashCopy using the DS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8.4.1 Remote FlashCopy commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8.4.2 Parameters used in remote FlashCopy commands . . . . . . . . . . . . . . . . . . . . . . . 79
8.5 FlashCopy management using the DS SM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8.5.1 Initiate FlashCopy using Create . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8.5.2 Display properties of existing FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
iv IBM System Storage DS8000 Series: Copy Services in Open Environments

8.5.3 Reverse existing FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
8.5.4 Initiate background copy for a persistent FlashCopy relationship. . . . . . . . . . . . . 87
8.5.5 Resynchronize target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
8.5.6 Delete existing FlashCopy relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Chapter 9. FlashCopy performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

9.1 FlashCopy performance overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
9.1.1 Distribution of the workload - source and target volumes location . . . . . . . . . . . . 92
9.1.2 LSS/LCU versus rank considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
9.1.3 Rank geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
9.1.4 Incremental FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
9.2 FlashCopy establish performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
9.3 Background copy performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
9.4 FlashCopy impact to applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
9.5 FlashCopy options - considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
9.6 FlashCopy scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
9.6.1 Scenario #1: Back up to disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
9.6.2 Scenario #2: Back up to tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
9.6.3 Scenario #3: FlashCopy during peak application activity . . . . . . . . . . . . . . . . . . . 97
9.6.4 Scenario #4: Ranks reserved for FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Chapter 10. FlashCopy examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

10.1 Create test system or integration system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
10.1.1 One time test system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
10.1.2 Multiple setup of a test system with the same content . . . . . . . . . . . . . . . . . . . 102
10.2 Create backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
10.2.1 Create a FlashCopy for backup purposes without volume copy . . . . . . . . . . . . 103
10.2.2 Incremental FlashCopy for backup purposes . . . . . . . . . . . . . . . . . . . . . . . . . . 104
10.2.3 Using a target volume to restore its contents back to the source . . . . . . . . . . . 104
Part 4. Metro Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Chapter 11. Metro Mirror overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

11.1 Metro Mirror overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
11.2 Metro Mirror volume state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
11.3 Data consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
11.4 Rolling disaster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
11.5 Automation and management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Chapter 12. Metro Mirror options and configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 113

12.1 Basic Metro Mirror operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
12.2 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
12.3 Failover and Failback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
12.4 Consistency Group function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
12.4.1 Data consistency and dependent writes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
12.4.2 Consistency Group function - how it works . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
12.5 Metro Mirror paths and links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
12.5.1 Fibre Channel links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
12.5.2 Logical paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
12.6 Bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
12.7 LSS design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
12.8 Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
12.9 Symmetrical configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
12.10 Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Contents v
12.11 Hardware requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Chapter 13. Metro Mirror performance and scalability . . . . . . . . . . . . . . . . . . . . . . . . 129

13.1 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
13.1.1 Initial synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
13.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
Chapter 14. Metro Mirror interfaces and examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

14.1 Metro Mirror interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
14.2 Copy Services network components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
14.3 DS Command-Line Interface (DS CLI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
14.4 DS Storage Manager GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
14.5 Set up Metro Mirror environment using DS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
14.5.1 Preparing to work with the DS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
14.5.2 Set up of Metro Mirror configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
14.5.3 Determine the available Fibre Channel links. . . . . . . . . . . . . . . . . . . . . . . . . . . 139
14.5.4 Create Metro Mirror paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
14.5.5 Create Metro Mirror pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
14.6 Remove Metro Mirror environment using DS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
14.6.1 Remove Metro Mirror pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
14.6.2 Remove paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
14.7 Manage the Metro Mirror environment with the DS CLI . . . . . . . . . . . . . . . . . . . . . . 144
14.7.1 Suspend and resume Metro Mirror data transfer . . . . . . . . . . . . . . . . . . . . . . . 144
14.7.2 Add and remove paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
14.8 Failover and Failback functions for sites switching . . . . . . . . . . . . . . . . . . . . . . . . . . 147
14.8.1 Metro Mirror Failover function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
14.8.2 Metro Mirror Failback function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
14.9 Freezepprc and unfreezepprc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
14.10 DS Storage Manager GUI examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
14.10.1 Create paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
14.10.2 Add paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
14.10.3 Change options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
14.10.4 Delete paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
14.10.5 Create volume pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
14.10.6 Suspend volume pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
14.10.7 Resume volume pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
14.10.8 Metro Mirror Failover. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
14.10.9 Metro Mirror Failback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Part 5. Global Copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Chapter 15. Global Copy overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

15.1 Global Copy overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
15.2 Volume states and change logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
15.3 Global Copy positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
Chapter 16. Global Copy options and configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 191

16.1 Global Copy basic options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
16.1.1 Establish Global Copy pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
16.1.2 Suspend Global Copy Pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
16.1.3 Resume Global Copy Pair. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
16.1.4 Terminate Global Copy Pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
16.1.5 Converting a Global Copy pair to Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . 193
16.2 Creating a consistent point-in-time copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
vi IBM System Storage DS8000 Series: Copy Services in Open Environments

16.2.1 Procedure to take a consistent point-in-time copy . . . . . . . . . . . . . . . . . . . . . . 194
16.2.2 Making a FlashCopy at the remote site. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
16.3 Hardware requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
16.4 Global Copy connectivity - paths and links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
16.4.1 Global Copy Fibre Channel links. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
16.4.2 Logical paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
16.5 Bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
16.6 LSS design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
16.7 Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
16.8 DS8000 configuration at the remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Chapter 17. Global Copy interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

17.1 Global Copy interfaces - overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
17.2 Copy Services network components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
17.3 DS Command-Line Interface (DS CLI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
17.3.1 Global Copy path commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
17.3.2 Global Copy pair commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
17.4 DS Storage Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
17.4.1 Path panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
17.4.2 Metro Mirror panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Chapter 18. Global Copy performance and scalability . . . . . . . . . . . . . . . . . . . . . . . . 211

18.1 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
18.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
18.2.1 Adding capacity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
Chapter 19. Global Copy examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

19.1 Set up Global Copy environment using DS-CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
19.1.2 Set up of Global Copy configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
19.1.3 Determine the available Fibre Channel links. . . . . . . . . . . . . . . . . . . . . . . . . . . 215
19.1.4 Create paths for Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
19.1.5 Create Global Copy pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
19.2 Remove Global Copy environment using DS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
19.2.1 Remove Global Copy pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
19.2.2 Remove paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
19.3 Maintain Global Copy environment using DS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
19.3.1 Suspend and resume Global Copy data transfer . . . . . . . . . . . . . . . . . . . . . . . 220
19.3.2 Change copy mode from Global Copy to Metro Mirror . . . . . . . . . . . . . . . . . . . 221
19.3.3 Change the copy mode from Metro Mirror to Global Copy . . . . . . . . . . . . . . . . 223
19.3.4 Add and remove paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
19.4 Periodic off-site backup procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
19.4.1 Initial setup for this environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
19.4.2 Periodical backup operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
19.5 DS Storage Manager GUI examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
19.5.1 Establish paths with the DS Storage Manager GUI . . . . . . . . . . . . . . . . . . . . . 230
19.5.2 Establish Global Copy pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
19.5.3 Monitoring the copy status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
19.5.4 Convert to Metro Mirror (synchronous) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
19.5.5 Suspend a pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Part 6. Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Chapter 20. Global Mirror overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Contents vii
20.1 Synchronous and non synchronous data replication . . . . . . . . . . . . . . . . . . . . . . . . 244
20.1.1 Synchronous data replication and dependent writes . . . . . . . . . . . . . . . . . . . . 244
20.1.2 Asynchronous data replication and dependent writes. . . . . . . . . . . . . . . . . . . . 248
20.2 Basic concepts of Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
20.3 Set up a Global Mirror session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
20.3.1 Simple configuration to start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
20.3.2 Establish connectivity to remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
20.3.3 Create Global Copy relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
20.3.4 Introduce FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
20.3.5 Define the Global Mirror session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
20.3.6 Populate the Global Mirror session with volumes . . . . . . . . . . . . . . . . . . . . . . . 258
20.3.7 Start the Global Mirror session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
20.4 Consistency Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
20.4.1 Consistency Group formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
20.4.2 Consistency Group parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Chapter 21. Global Mirror options and configuration . . . . . . . . . . . . . . . . . . . . . . . . . 263

21.1 Terminology used in Global Mirror environments . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
21.2 Create a Global Mirror environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
21.3 Modify a Global Mirror session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
21.3.1 Add or remove volumes to the Global Mirror session . . . . . . . . . . . . . . . . . . . . 267
21.3.2 Add or remove storage disk subsystems or LSSs . . . . . . . . . . . . . . . . . . . . . . 268
21.3.3 Modify the Global Mirror session parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 268
21.3.4 Global Mirror environment topology changes . . . . . . . . . . . . . . . . . . . . . . . . . . 269
21.3.5 Remove FlashCopy relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
21.3.6 Remove the Global Mirror environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
21.4 Global Mirror with multiple storage disk subsystems . . . . . . . . . . . . . . . . . . . . . . . . 270
21.5 Recovery scenario after production site failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
21.5.1 Normal Global Mirror operation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
21.5.2 Production site failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
21.5.3 Global Copy Failover from B to A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
21.5.4 Verify for valid Consistency Group state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
21.5.5 Set consistent data on B volumes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
21.5.6 Re-establish FlashCopy relationships between B and C . . . . . . . . . . . . . . . . . 282
21.5.7 Restart the application at the remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
21.5.8 Prepare to switch back to the local site. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
21.5.9 Return to the local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
21.5.10 Conclusions of failover/failback example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
Chapter 22. Global Mirror interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

22.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
22.2 DS Command-Line Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
22.3 DS Storage Manager GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
22.4 eRCMF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
Chapter 23. Global Mirror performance and scalability. . . . . . . . . . . . . . . . . . . . . . . . 295

23.1 Performance aspects for Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
23.2 Performance considerations at coordination time . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
23.3 Consistency Group drain time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
23.4 Remote storage disk subsystem configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
23.5 Balancing the disk subsystem configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
23.6 Growth within Global Mirror configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Chapter 24. Global Mirror examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
viii IBM System Storage DS8000 Series: Copy Services in Open Environments
24.1 Set up a Global Mirror environment using the DS CLI . . . . . . . . . . . . . . . . . . . . . . . 306
24.1.2 Configuration used for the environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
24.1.3 Setup procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
24.1.4 Create Global Copy relationships - A to B volumes . . . . . . . . . . . . . . . . . . . . . 307
24.1.5 Create FlashCopy relationships - B to C volumes . . . . . . . . . . . . . . . . . . . . . . 308
24.1.6 Start Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
24.2 Remove a Global Mirror environment with the DS CLI . . . . . . . . . . . . . . . . . . . . . . . 315
24.2.1 End Global Mirror processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
24.2.2 Remove the A volumes from the Global Mirror session . . . . . . . . . . . . . . . . . . 317
24.2.3 Remove the Global Mirror session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
24.2.4 Terminate FlashCopy pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
24.2.5 Terminate Global Copy pairs and remove the paths. . . . . . . . . . . . . . . . . . . . . 319
24.3 Manage the Global Mirror environment with the DS CLI. . . . . . . . . . . . . . . . . . . . . . 320
24.3.1 Pause and resume Global Mirror Consistency Group formation. . . . . . . . . . . . 320
24.3.2 Change the Global Mirror tuning parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 323
24.3.3 Stop and start Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
24.3.4 Add and remove A volumes to the Global Mirror environment . . . . . . . . . . . . . 325
24.3.5 Add and remove an LSS to an existing Global Mirror environment . . . . . . . . . 327
24.3.6 Add and remove a subordinate disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . 329
24.4 Recovery scenario after local site failure with the DS CLI. . . . . . . . . . . . . . . . . . . . . 330
24.4.1 Stop Global Mirror processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
24.4.2 Perform Global Copy Failover from B to A . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
24.4.3 Verify for valid Consistency Group state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
24.4.4 Reverse FlashCopy from B to C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
24.4.5 Re-establish the FlashCopy relationship from B to C . . . . . . . . . . . . . . . . . . . . 337
24.4.6 Restart the application at the remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
24.5 Return to the local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
24.5.1 Create paths from B to A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
24.5.2 Perform Global Copy Failback from B to A . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
24.5.3 Query for the Global Copy first pass completion. . . . . . . . . . . . . . . . . . . . . . . . 341
24.5.4 Quiesce the application at the remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
24.5.5 Query the Out Of Sync Tracks until it shows zero . . . . . . . . . . . . . . . . . . . . . . 342
24.5.6 Create paths from A to B if they do not exist. . . . . . . . . . . . . . . . . . . . . . . . . . . 342
24.5.7 Perform Global Copy Failover from A to B . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
24.5.8 Perform Global Copy Failback from A to B . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
24.5.9 Start Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
24.5.10 Start the application at the local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
24.6 Practice disaster recovery readiness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
24.6.1 Query the Global Mirror environment to look at the situation . . . . . . . . . . . . . . 348
24.6.2 Pause Global Mirror and check its completion . . . . . . . . . . . . . . . . . . . . . . . . . 348
24.6.3 Pause Global Copy pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
24.6.4 Perform Global Copy Failover from B to A . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
24.6.5 Create consistent data on B volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
24.6.6 Wait for the FlashCopy background copy to complete . . . . . . . . . . . . . . . . . . . 350
24.6.7 Re-establish the FlashCopy relationships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
24.6.8 Take FlashCopy from B to D. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
24.6.9 Perform the disaster recovery testing using the D volume . . . . . . . . . . . . . . . . 352
24.6.10 Perform Global Copy Failback from A to B . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
24.6.11 Wait for the Global Copy first pass to complete . . . . . . . . . . . . . . . . . . . . . . . 354
24.6.12 Resume Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354
24.7 DS Storage Manager GUI examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
24.8 Setup a Global Mirror environment with the DS GUI. . . . . . . . . . . . . . . . . . . . . . . . . 355
Contents ix
24.8.1 Define paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
24.8.2 Create Global Copy pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
24.8.3 Create FlashCopy relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
24.8.4 Create the Global Mirror session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
24.9 Manage the Global Mirror environment with the DS GUI . . . . . . . . . . . . . . . . . . . . . 369
24.9.1 View settings and error information of the Global Mirror session . . . . . . . . . . . 370
24.9.2 View information of volumes in the Global Mirror session . . . . . . . . . . . . . . . . 371
24.9.3 Pause a Global Mirror session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
24.9.4 Resume a Global Mirror session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
24.9.5 Modify a Global Mirror session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Part 7. Metro/Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
Chapter 25. Metro/Global Mirror overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

25.1 Metro/Global Mirror overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
25.1.1 Metro/Global Mirror design objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
25.2 Metro/Global Mirror processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
Chapter 26. Configuration and setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381

26.1 Metro/Global Mirror configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
26.1.1 Metro/Global Mirror with additional Global Mirror . . . . . . . . . . . . . . . . . . . . . . . 382
26.1.2 Metro/Global Mirror with multiple storage subsystems . . . . . . . . . . . . . . . . . . . 383
26.2 Configuration examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
26.3 Initial setup of Metro/Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
26.3.1 Identifying the PPRC ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
26.3.2 Step 1: Set up all Metro Mirror and Global Mirror paths . . . . . . . . . . . . . . . . . . 386
26.3.3 Step 2: Set up Global Copy NOCOPY from intermediate to remote sites . . . . 388
26.3.4 Step 3: Set up Metro Mirror between local and intermediate sites . . . . . . . . . . 388
26.3.5 Step 4: Set up FlashCopy at remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
26.3.6 Step 5: Create Global Mirror session and add volumes to session . . . . . . . . . 389
26.3.7 Step 6: Start Global Mirror at intermediate site . . . . . . . . . . . . . . . . . . . . . . . . . 390
26.4 Going from Metro Mirror to Metro/Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
26.5 Recommendations for setting up Metro/Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . 393
Chapter 27. General Metro/Global Mirror operations. . . . . . . . . . . . . . . . . . . . . . . . . . 395

27.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
27.2 General consideration for storage failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
27.3 Check pair status before failover. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
27.4 Freeze and unfreeze Metro Mirror volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
27.5 Suspend volumes before failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
27.6 Remove volumes from the session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
27.7 Check consistency at the remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
27.8 Set up an additional Global Mirror from remote site . . . . . . . . . . . . . . . . . . . . . . . . . 403
27.8.1 Step 1: Create Global Copy from remote to intermediate site . . . . . . . . . . . . . 404
27.8.2 Step 2: Create FlashCopy at the intermediate site . . . . . . . . . . . . . . . . . . . . . . 405
27.8.3 Step 3: Create session and Global Mirror at remote site . . . . . . . . . . . . . . . . . 406
Chapter 28. Planned recovery scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407

28.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
28.2 Recovery at intermediate site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
28.2.1 Step 1: Stop the production application at the local site . . . . . . . . . . . . . . . . . . 409
28.2.2 Step 2: Suspend the Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
28.2.3 Step 3: Failover the intermediate site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
28.2.4 Step 4: Start the production application at the intermediate site. . . . . . . . . . . . 411
x IBM System Storage DS8000 Series: Copy Services in Open Environments

28.3 Return to local from intermediate site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
28.3.1 Step 1: Stop I/O at the intermediate site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
28.3.2 Step 2: Terminate Global Mirror or remove volumes from the session. . . . . . . 413
28.3.3 Step 3: Suspend Global Copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
28.3.4 Step 4: Fail back Metro Mirror to local site and wait for Full Duplex . . . . . . . . . 414
28.3.5 Steps 5 and 6: Suspend Metro Mirror and fail over to the local site . . . . . . . . . 414
28.3.6 Step 7: Fail back Metro Mirror from the local site to the intermediate site . . . . 415
28.3.7 Step 8: Resume Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
28.3.8 Step 9: Start I/O at the local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
28.3.9 Step 10: Start Global Mirror or add volumes to the session . . . . . . . . . . . . . . . 416
28.4 Recovery at remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
28.4.1 Step 1: Stop I/O at the local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
28.4.2 Step 2: Terminate Global Mirror or remove volumes from session. . . . . . . . . . 418
28.4.3 Step 3: Terminate Global Copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
28.4.4 Step 4: Suspend Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
28.4.5 Step 5: Failover Metro Mirror to intermediate site . . . . . . . . . . . . . . . . . . . . . . . 419
28.4.6 Step 6: Establish Global Copy from remote to intermediate site. . . . . . . . . . . . 420
28.4.7 Step 7: Start I/O at the remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420
28.5 Return from remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
28.5.1 Step 1: Stop I/O at remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
28.5.2 Step 2: Fail back Metro Mirror from the intermediate site to the local site . . . . 422
28.5.3 Step 3: Terminate Global Copy from remote to intermediate site . . . . . . . . . . . 422
28.5.4 Step 4: Suspend Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
28.5.5 Step 5: Fail over to local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
28.5.6 Step 6: Fail back Metro Mirror from the local site to the intermediate site . . . . 424
28.5.7 Step 7: Create Global Copy from intermediate to remote site . . . . . . . . . . . . . 424
28.5.8 Step 8: Start I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
28.5.9 Step 9: Start Global Mirror or add volumes to the session . . . . . . . . . . . . . . . . 425
Chapter 29. Disaster recovery test scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427

29.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
29.2 Disaster recovery test at the intermediate site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
29.2.1 Step 1: Prepare the failover for disaster recovery test . . . . . . . . . . . . . . . . . . . 429
29.2.2 Step 2: Set up FlashCopy to the additional volumes . . . . . . . . . . . . . . . . . . . . 430
29.2.3 Step 3: Set up PPRC paths from the local site to the intermediate site . . . . . . 431
29.2.4 Step 4: Resume Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
29.2.5 Step 5: Start I/O on the disaster recovery host . . . . . . . . . . . . . . . . . . . . . . . . . 432
29.3 Disaster recovery test at remote site. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
Chapter 30. Unplanned scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435

30.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
30.2 Server outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
30.3 Link failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
30.3.1 Metro Mirror link failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
30.3.2 Global Copy link failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
30.4 Partial disasters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
30.5 Data center outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Chapter 31. MGM Incremental Resync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443

31.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
31.1.1 Functional description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
31.1.2 Options for DSCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
31.2 Setting up Metro/Global Mirror with Incremental Resync . . . . . . . . . . . . . . . . . . . . . 447
31.2.1 Setup of Incremental Resync Metro/Global Mirror . . . . . . . . . . . . . . . . . . . . . . 447
Contents xi
31.2.2 Going from Global Mirror to Incremental Resync Metro/Global Mirror . . . . . . . 448
31.3 Failure at the local site scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
31.3.1 Local site fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455
31.3.2 Local site is back. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
31.3.3 Returning to the original configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
31.4 Failure at intermediate site scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
31.4.1 Intermediate site failure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
31.4.2 Intermediate site back up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477
31.4.3 Re-synchronization at intermediate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
31.4.4 Restore original configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
Part 8. Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489
Chapter 32. Data migration through double cascading. . . . . . . . . . . . . . . . . . . . . . . . 491

32.1 Data migration with double cascading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
Chapter 33. Interoperability between DS6000 and DS8000 . . . . . . . . . . . . . . . . . . . . . 495

33.1 DS8000 and DS6000 Copy Services interoperability . . . . . . . . . . . . . . . . . . . . . . . . 496
33.2 Preparing the environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
33.2.1 Minimum microcode levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
33.2.2 Hardware and licensing requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
33.2.3 Network connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
33.2.4 Create matching user IDs and passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
33.2.5 Updating the DS CLI profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
33.2.6 Adding the Storage Complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
33.2.7 Volume size considerations for Remote Mirror Copy . . . . . . . . . . . . . . . . . . . . 501
33.2.8 Establishment errors on newly created volumes. . . . . . . . . . . . . . . . . . . . . . . . 503
33.3 RMC: Establishing paths between DS6000 and DS8000 . . . . . . . . . . . . . . . . . . . . . 503
33.3.1 Decoding port IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
33.3.2 Path creation using the DS GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
33.3.3 Establish logical paths between DS8000 and DS6000 using DS CLI. . . . . . . . 504
33.4 Managing Metro Mirror or Global Copy pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506
33.4.1 Managing Metro Mirror or Global Copy pairs using the DS GUI . . . . . . . . . . . . 506
33.4.2 Managing Metro Mirror pairs using the DS CLI. . . . . . . . . . . . . . . . . . . . . . . . . 506
33.4.3 Managing Global Copy pairs using DS CLI. . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
33.5 Managing DS6000 to DS8000 Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
33.5.1 Managing Global Mirror pairs using the DS CLI . . . . . . . . . . . . . . . . . . . . . . . . 508
33.6 Managing DS6000 and DS8000 FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509
33.6.1 Creating a remote FlashCopy on a DS6000 using the DS CLI . . . . . . . . . . . . . 509
Chapter 34. Interoperability between DS8000 and ESS 800 . . . . . . . . . . . . . . . . . . . . 511

34.1 DS8000 and ESS 800 Copy Services interoperability . . . . . . . . . . . . . . . . . . . . . . . 512
34.2 Preparing the environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
34.2.1 Minimum microcode levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
34.2.2 Hardware and licensing requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
34.2.3 Network connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
34.2.4 Create matching user IDs and passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
34.2.5 Updating the DS CLI profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
34.2.6 Adding the Copy Services Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
34.2.7 Volume size considerations for RMC (PPRC). . . . . . . . . . . . . . . . . . . . . . . . . . 515
34.2.8 Volume address considerations on the ESS 800 . . . . . . . . . . . . . . . . . . . . . . . 519
34.2.9 Establishment errors on newly created volumes. . . . . . . . . . . . . . . . . . . . . . . . 519
34.3 RMC: Establishing paths between DS8000 and ESS 800 . . . . . . . . . . . . . . . . . . . . 519
34.3.1 Decoding port IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
xii IBM System Storage DS8000 Series: Copy Services in Open Environments
34.3.2 Path creation using the DS GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
34.3.3 Establish logical paths between DS8000 and ESS 800 using DS CLI . . . . . . . 523
34.4 Managing Metro Mirror or Global Copy pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
34.4.1 Managing Metro Mirror or Global Copy pairs using the DS GUI . . . . . . . . . . . . 526
34.4.2 Managing Metro Mirror pairs using the DS CLI. . . . . . . . . . . . . . . . . . . . . . . . . 530
34.4.3 Managing Global Copy pairs using the DS CLI. . . . . . . . . . . . . . . . . . . . . . . . . 531
34.5 Managing ESS 800 Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531
34.5.1 Managing Global Mirror pairs using the DS CLI . . . . . . . . . . . . . . . . . . . . . . . . 532
34.6 Managing ESS 800 FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
34.6.1 Creating an ESS 800 FlashCopy using the DS GUI . . . . . . . . . . . . . . . . . . . . . 533
34.6.2 Creating an ESS 800 FlashCopy using DS CLI . . . . . . . . . . . . . . . . . . . . . . . . 534
34.6.3 Creating a remote FlashCopy on an ESS 800 using DS CLI . . . . . . . . . . . . . . 535
Part 9. Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
Chapter 35. IBM TotalStorage Productivity Center for Replication . . . . . . . . . . . . . . 539

35.1 IBM TotalStorage Productivity Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540
35.2 Where we are coming from . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541
35.3 What TPC for Replication provides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541
35.4 Copy Services terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542
35.4.1 FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542
35.4.2 Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
35.4.3 Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
35.4.4 Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
35.4.5 Metro/Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
35.4.6 Failover/failback terminology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544
35.5 TPC for Replication terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545
35.5.1 TPC for Replication copy set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545
35.5.2 TPC for Replication session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
35.6 TPC for Replication session types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549
35.6.1 TPC for Replication Basic Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549
35.6.2 TPC for Replication Advanced Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549
35.7 TPC for Replication session states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550
35.8 Volumes in a copy set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550
35.8.1 Host volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
35.8.2 Target volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
35.8.3 Journal volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
35.9 TPC for Replication and scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
35.10 TPC for Replication system and connectivity overview. . . . . . . . . . . . . . . . . . . . . . 552
35.11 TPC for Replication monitoring and freeze capability . . . . . . . . . . . . . . . . . . . . . . . 558
35.12 TPC for Replication heartbeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559
35.13 Supported platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560
35.14 Hardware requirements for TPC for Replication servers. . . . . . . . . . . . . . . . . . . . . 560
35.15 TPC for Replication GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
35.15.1 Connect to the TPC for Replication GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563
35.15.2 Health Overview panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564
35.15.3 Sessions panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565
35.15.4 Storage Subsystems panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
35.15.5 Path Management panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569
35.15.6 RM Server Configuration panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
35.15.7 Advanced Tools panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
35.15.8 Console log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
35.16 Command line interface to TPC for Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
Contents xiii
Chapter 36. IBM TotalStorage Rapid Data Recovery for UNIX and Windows . . . . . . 575
36.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
36.1.1 Solution highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
36.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
36.3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
36.4 Additional information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
Chapter 37. IBM TotalStorage Continuous Availability for Windows . . . . . . . . . . . . . 585

37.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
37.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
37.3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
37.3.1 Service and ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589
Appendix A. Open systems specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591

AIX specifics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
AIX and FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
AIX and Remote Mirror and Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596
Windows and Remote Mirror and Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598
Copy services with Windows volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
Microsoft Volume Shadow Copy Services (VSS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
Microsoft Virtual Disk Service (VDS). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604
SUN Solaris and Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
FlashCopy without a volume manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
Remote copy without a Volume Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
Copy Services using VERITAS Volume Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
HP-UX and Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
HP-UX and FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
HP-UX with Remote Mirror and Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
VMware ESX server and Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616
VMware ESX server and FlashCopy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
VMware ESX server with Remote Mirror and Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . 626
Appendix B. Copy Services with System i5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 630
System i5 functions and external storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 630
System i5 structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
Single-level storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
Input Output Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632
Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
Independent Auxiliary Storage Pools (IASPs). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635
Metro Mirror for an IASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
Solution description. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637
Solution benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 638
Planning and requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 639
Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
Implementation and usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 641
Global Mirror for an IASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667
Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 670
Metro Mirror for the entire disk space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 672
xiv IBM System Storage DS8000 Series: Copy Services in Open Environments
Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674
Global Mirror for the entire disk space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 682
Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684
FlashCopy of IASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689
Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 691
Using Metro Mirror and FlashCopy of IASP in the same scenario . . . . . . . . . . . . . . . . 692
FlashCopy of the entire disk space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695
Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 697
Appendix C. Simple Network Management Protocol (SNMP) . . . . . . . . . . . . . . . . . . . 701

SNMP overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 702
Physical connection events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 702
Remote copy events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705
Global Mirror related SNMP traps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705
Appendix D. Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709

Licenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 710
Authorized level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713
Charging example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713
Appendix E. CLI migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715

Migrating ESS CLI to DS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716
Review the ESS tasks to migrate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716
Convert the individual tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 718
Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723
Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723
Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724
How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725
Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727
Contents xv
xvi IBM System Storage DS8000 Series: Copy Services in Open Environments
Figures
2-1 Logical view of an Storage Complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2-2 Network Interface communication structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2-3 Remote Mirror and Copy between Storage Complexes . . . . . . . . . . . . . . . . . . . . . . 13
3-1 A single Storage Complex with a single Storage Unit . . . . . . . . . . . . . . . . . . . . . . . . 19
3-2 Add Storage Complex panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3-3 Successful addition of a Storage Complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3-4 Help option location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3-5 FlashCopy help main page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5-1 FlashCopy operational environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5-2 FlashCopy at time t0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5-3 Reads from source and target volumes and writes to source volume . . . . . . . . . . . . 40
5-4 Writes to target volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5-5 Target volume after full volume FlashCopy relationship finished. . . . . . . . . . . . . . . . 42
5-6 FlashCopy after updates to the target volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5-7 FlashCopy and Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5-8 FlashCopy and Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5-9 FlashCopy and Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5-10 DS8300 Storage Facility Image (SFI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6-1 Multiple Relationship FlashCopy possibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6-2 FlashCopy target is Metro Mirror (or Global Copy) primary . . . . . . . . . . . . . . . . . . . . 49
6-3 Updates to the target volume caused by a refresh target FlashCopy . . . . . . . . . . . . 50
6-4 FlashCopy with Start Change Recording set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6-5 Refresh of the FlashCopy target volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6-6 Remote FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6-7 Reverse restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6-8 FlashCopy options and interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
8-1 Overview of parameters used in DS CLI FlashCopy commands. . . . . . . . . . . . . . . . 67
8-2 DS CLI remote FlashCopy commands and parameters . . . . . . . . . . . . . . . . . . . . . . 79
8-3 FlashCopy Create with DS SM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8-4 Common option window for FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
8-5 FlashCopy display properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
8-6 General folder with FlashCopy properties information. . . . . . . . . . . . . . . . . . . . . . . . 83
8-7 Display out-of-synch tracks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
8-8 Parameters or copy options to use with reverse action . . . . . . . . . . . . . . . . . . . . . . . 86
8-9 Initiate background copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
8-10 Prompt window for background copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
8-11 Resync target select option to re-synchronize FlashCopy relationship . . . . . . . . . . . 88
8-12 Prompt window to detail resync request for FlashCopy relationship . . . . . . . . . . . . . 88
8-13 Delete select option to delete FlashCopy relationship . . . . . . . . . . . . . . . . . . . . . . . . 89
8-14 Prompt window to confirm delete request for FlashCopy relationship . . . . . . . . . . . . 89
11-1 Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
12-1 Metro Mirror Failover and Failback sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
12-2 Consistency Group: Example1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
12-3 Consistency Group: Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
12-4 Consistency Group: Example.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
12-5 Logical paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
12-6 Logical paths over a physical link for Metro Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . 123
12-7 Symmetrical Metro Mirror configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
© Copyright IBM Corp. 2005, 2006. All rights reserved. xvii

14-1 DS8000 Copy Services network components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
14-2 DS8000 configuration in the Metro Mirror setup example . . . . . . . . . . . . . . . . . . . . 137
14-3 Metro Mirror environment to be set up. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
14-4 Metro Mirror environment for sites switch example . . . . . . . . . . . . . . . . . . . . . . . . . 148
14-5 Create path - Paths panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
14-6 Create path - Step 1: select source LSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
14-7 Create path - Step 2: select target LSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
14-8 Create path - Step 3: select source I/O ports. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
14-9 Create path - Step 4: select target I/O ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
14-10 Create path - Step 5: select path options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
14-11 Create path - Step 6: verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
14-12 Create path - result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
14-13 Add paths - Paths panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
14-14 Add paths - Step 1: select source LSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
14-15 Add paths - Step 2: select target LSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
14-16 Add paths - Step 3: select source I/O ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
14-17 Add paths - Step 4: select target I/O port. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
14-18 Add paths - Step 5: select path options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
14-19 Add paths - Step 6: verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
14-20 Add paths - Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
14-21 Path panel - LSS copy options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
14-22 LSS copy options panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
14-23 Delete paths - Paths panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
14-24 Delete paths warning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
14-25 Delete paths - Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
14-26 Create MM pair - Metro Mirror panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
14-27 Create MM pair - Step 1: volume pairing method . . . . . . . . . . . . . . . . . . . . . . . . . . 172
14-28 Create MM pair - Step 2: select source volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
14-29 Create MM pair - Step 3: select target volumes (Auto pairing) . . . . . . . . . . . . . . . . 173
14-30 Create MM pair - Step 4: select copy options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
14-31 Create MM pair - Step 5: verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
14-32 Create MM pair - Volume pairs created and in full duplex state . . . . . . . . . . . . . . . 175
14-33 Suspend - Metro Mirror panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
14-34 Suspend - Suspend at source or target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
14-35 Suspend - Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
14-36 Resume - Metro Mirror panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
14-37 Resume - Verify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
14-38 Resume - Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
14-39 Failover - Metro Mirror panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
14-40 Failover verify. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
14-41 Result of a failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
14-42 Failback - Metro Mirror panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
14-43 Failback - Verify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
14-44 Failback - Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
14-45 Failback - View from the production site Storage Unit . . . . . . . . . . . . . . . . . . . . . . . 183
14-46 Starting failover - View on production site DS8000 . . . . . . . . . . . . . . . . . . . . . . . . . 183
14-47 Starting failback on the production site DS8000 . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
14-48 Production site storage unit at conclusion of disaster recovery exercise. . . . . . . . . 184
15-1 Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
15-2 Global Copy and Mirror state change logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
15-3 Global Copy positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
16-1 Create consistent data in Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
16-2 FlashCopy establishment from the production site . . . . . . . . . . . . . . . . . . . . . . . . . 195
xviii IBM System Storage DS8000 Series: Copy Services in Open Environments
17-1 Paths panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
17-2 LSS Copy Options panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
17-3 Metro Mirror panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
19-1 DS8000 configuration in the Global Copy set up example . . . . . . . . . . . . . . . . . . . 214
19-2 Global Copy environment to set up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
19-3 The DS8000 environment for Global Copy offsite backup. . . . . . . . . . . . . . . . . . . . 224
19-4 Global Copy offsite backup scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
19-5 Paths panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
19-6 Select source LSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
19-7 Select target LSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
19-8 Selecting source ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
19-9 Select target I/O ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
19-10 Select path options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
19-11 Verification panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
19-12 Paths defined . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
19-14 Volume Paring Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
19-15 Select source volumes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
19-16 Select target volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
19-17 Copy options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
19-18 Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
19-19 Global Copy volumes in copy pending status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
19-20 Out of sync tracks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
19-21 Selecting the action to convert to synchronous . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
19-22 Convert to synchronous confirmation panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
20-1 Synchronous data replication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
20-2 Synchronous data replication and unnecessary recovery after local site failure . . . 245
20-3 Synchronous data replication, freeze, and restart without recovery required . . . . . 247
20-4 Asynchronous data replication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
20-5 Global Copy sequence of data arrival not preserved at remote site . . . . . . . . . . . . 249
20-6 Global Copy non synchronous data replication involving multiple disk subsystems 250
20-7 Distributed application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
20-8 Global Mirror as a distributed application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
20-9 Start with simple application environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
20-10 Establish Global Copy connectivity between both sites. . . . . . . . . . . . . . . . . . . . . . 254
20-11 Establish Global Copy volume pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
20-12 Introducing FlashCopy in the Global MIrror solution . . . . . . . . . . . . . . . . . . . . . . . . 256
20-13 Define Global Mirror session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
20-14 Add Global Copy source volumes to Global Mirror session. . . . . . . . . . . . . . . . . . . 258
20-15 Start Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
20-16 Formation of consistent set of volumes at the remote site. . . . . . . . . . . . . . . . . . . . 260
20-17 Consistency Group tuning parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
21-1 Global Mirror basic configuration with master and subordinate disk subsystems . . 266
21-2 Define Global Mirror session to all potentially involved storage disk subsystems . . 270
21-3 Decide for master disk subsystem and start Global Mirror session . . . . . . . . . . . . . 271
21-4 Global Mirror paths over FCP links between source storage disk subsystems . . . . 272
21-5 Dedicated and shared links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
21-6 Dedicated Global Mirror links and dedicated Global Copy links . . . . . . . . . . . . . . . 274
21-7 Normal Global Mirror operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
21-8 Production site fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Figures xix
21-9 Perform Global Copy Failover from B to A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
21-10 Check Consistency Group state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
21-11 FlashCopy consistency group creation process . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
21-12 Set a consistent set of B volumes using the C volumes as source . . . . . . . . . . . . . 281
21-13 Restart applications at remote site. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
21-14 Failback operation from B to A in preparation for returning to local site . . . . . . . . . 283
21-15 Global Copy Failover from A to B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
21-16 Global Copy failback from A to B and resync Global Copy volumes . . . . . . . . . . . . 284
21-17 Establish Global Mirror FlashCopy relationship between B and C . . . . . . . . . . . . . 285
21-18 Start Global Mirror session and resume application I/O at local site . . . . . . . . . . . . 285
22-1 Global Mirror main panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
22-2 Global Mirror pull-down list - no session selected . . . . . . . . . . . . . . . . . . . . . . . . . . 292
22-3 Global Mirror pull-down list - Session selected . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
23-1 Global Copy with write hit at the remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
23-2 Global Copy with overcommitted NVS at the remote site . . . . . . . . . . . . . . . . . . . . 297
23-3 Coordination time: How it impacts application write I/Os . . . . . . . . . . . . . . . . . . . . . 298
23-4 Remote disk subsystem with all ranks containing equal numbers of volumes . . . . 300
23-5 Remote disk subsystem with D volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
24-1 DS8000 configuration in the Global Mirror example . . . . . . . . . . . . . . . . . . . . . . . . 306
24-2 Start Global Mirror session with a subordinate DS8000#3 . . . . . . . . . . . . . . . . . . . 314
24-3 Global Mirror example before unplanned production site failure . . . . . . . . . . . . . . . 330
24-4 Site swap scenario after failoverpprc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
24-5 Site swap scenario after the reverseflash command was issued . . . . . . . . . . . . . . 335
24-6 After the completion of the background copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
24-7 After mkflash B to C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
24-8 After the application start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
24-9 Site swap scenario after Global Copy Failback from B to A . . . . . . . . . . . . . . . . . . 340
24-10 Site swap scenario after Global Copy Failover from A to B . . . . . . . . . . . . . . . . . . . 343
24-11 Site swap scenario after failbackpprc A to B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
24-12 Start Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
24-13 Start the application at the local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
24-14 After mkflash B to D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
24-15 Perform Global Copy Failback A to B - test scenario. . . . . . . . . . . . . . . . . . . . . . . . 352
24-16 Global Copy paths creation - Launch the creation process . . . . . . . . . . . . . . . . . . . 356
24-17 Global Copy paths creation - step 1: select the source LSS . . . . . . . . . . . . . . . . . . 356
24-18 Global Copy paths creation - step 2: select the target LSS . . . . . . . . . . . . . . . . . . . 357
24-19 Global Copy paths creation - step 3: select the source I/O ports. . . . . . . . . . . . . . . 357
24-20 Global Copy paths creation - step 4: select the target I/O ports . . . . . . . . . . . . . . . 358
24-21 Global Copy paths creation - step 5: select the paths options. . . . . . . . . . . . . . . . . 358
24-22 Global Copy paths creation - step 6: verification . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
24-23 Global Copy creation - Launch the creation process . . . . . . . . . . . . . . . . . . . . . . . . 359
24-24 Global Copy creation - step 1: choose volume pairing method . . . . . . . . . . . . . . . . 360
24-25 Global Copy creation - step 2: select the source volumes. . . . . . . . . . . . . . . . . . . . 360
24-26 Global Copy creation - Step 3: select the target volume 1. . . . . . . . . . . . . . . . . . . . 361
24-27 Global Copy creation - Step 3: select the target volume 2. . . . . . . . . . . . . . . . . . . . 361
24-28 Global Copy creation - Step 4: select the copy options . . . . . . . . . . . . . . . . . . . . . . 362
24-29 Global Copy creation - Step 5: verification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
24-30 FlashCopy creation - Launch the creation process . . . . . . . . . . . . . . . . . . . . . . . . . 364
24-31 FlashCopy creation - Step 1: Select the relationship type . . . . . . . . . . . . . . . . . . . . 364
24-32 FlashCopy creation - Step 2: select the source volumes. . . . . . . . . . . . . . . . . . . . . 365
24-33 FlashCopy creation - Step 3: select the target volumes . . . . . . . . . . . . . . . . . . . . . 365
24-34 FlashCopy creation - Step 4: select the common option . . . . . . . . . . . . . . . . . . . . . 366
24-35 FlashCopy creation - Step 5: verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
xx IBM System Storage DS8000 Series: Copy Services in Open Environments

24-36 Global Mirror creation - Launch the creation process . . . . . . . . . . . . . . . . . . . . . . . 367
24-37 Global Mirror creation - Step 1: select volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
24-38 Global Mirror creation - Step 2: define properties . . . . . . . . . . . . . . . . . . . . . . . . . . 368
24-39 Global Mirror creation - Step 3 - Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
24-40 Global Mirror - Visualize the session status. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
24-41 View Global Mirror session’s properties: Launch the viewing process . . . . . . . . . . 370
24-42 View Global Mirror session settings: General tab . . . . . . . . . . . . . . . . . . . . . . . . . . 370
24-43 View Global Mirror session errors information - Failures tab . . . . . . . . . . . . . . . . . . 371
24-44 Global Mirror session volumes information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
24-45 Pause Global Mirror: Confirm the pause of the Global Mirror session. . . . . . . . . . . 372
24-46 Pause Global Mirror: View session status paused. . . . . . . . . . . . . . . . . . . . . . . . . . 372
24-47 Resume Global Mirror - Confirm the resume of the Global Mirror session . . . . . . . 373
25-1 Metro/Global Mirror elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
25-2 Metro/Global Mirror overview diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
26-1 Metro/Global Mirror with additional Global Mirror from remote to intermediate site. 382
26-2 Setup of a Metro/Global Mirror with multiple storage subsystems. . . . . . . . . . . . . . 383
26-3 Configuration example for an open systems environment. . . . . . . . . . . . . . . . . . . . 384
26-4 Set up Metro/Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
26-5 Migrating from Metro Mirror to Metro/Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . 393
27-1 Phases of consistency group formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
27-2 Set up an additional Global Mirror after failover to remote site . . . . . . . . . . . . . . . . 404
28-1 Recovery of production to the intermediate site. . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
28-2 Return from intermediate site to local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
28-3 Recovery of production to remote site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
28-4 Return from the remote to the local site. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
29-1 Failover to intermediate site for disaster recovery test . . . . . . . . . . . . . . . . . . . . . . 428
29-2 Failover for disaster recovery test to remote . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
30-1 Failover after server outage at local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
30-2 Link failure of the Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
30-3 Example for a partial disaster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
31-1 Setup of a Metro/Global Mirror for Incremental Resync. . . . . . . . . . . . . . . . . . . . . . 444
31-2 Incremental Resync phases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
31-3 Incremental Resync overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
31-4 Moving from Global Mirror to Incremental Resync Metro/Global Mirror. . . . . . . . . . 448
31-5 Local site failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455
31-6 Local site is back . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
31-7 Move incremental Resync Metro/Global Mirror back to local site . . . . . . . . . . . . . . 464
31-8 Failure at the intermediate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
31-9 Cleanup of the intermediate when it becomes available . . . . . . . . . . . . . . . . . . . . . 478
31-10 Re-synchronization of the intermediate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
31-11 Restoring the Metro/Global Mirror configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
32-1 Double cascading example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
32-2 Global Copy changed to Metro Mirror and direction reversed . . . . . . . . . . . . . . . . . 493
33-1 Add DS6000 complex to DS8000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499
33-2 Add DS8000 complex to DS6000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
33-3 DS6000 GUI showing multiple complexes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
34-1 Add 2105 Copy Services Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
34-2 Viewing ESS 800 volume size using ESS Specialist GUI . . . . . . . . . . . . . . . . . . . . 516
34-3 Displaying block count using Copy Services server . . . . . . . . . . . . . . . . . . . . . . . . 518
34-4 ESS 800 Port IDs decoded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
34-5 Path panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
34-6 Select source LSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
34-7 Select target LSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
Figures xxi
34-8 Select source I/O ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
34-9 Select target I/O ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
34-10 Path panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
34-11 Using ESS 800 Specialist GUI to display the ESS 800 WWNN . . . . . . . . . . . . . . . 524
34-12 Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
34-13 Pairing method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
34-14 Select source volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
34-15 First Metro Mirror target volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
34-16 Second Metro Mirror target volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529
34-17 Copy options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529
34-18 Managing Metro Mirror pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
34-19 Creating ESS 800 FlashCopy using DS GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533
34-20 Selecting ESS 800 source volumes for FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . 534
35-1 TPC software products suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540
35-2 Management of Copy Services Functions supported by TPC for Replication . . . . . 542
35-3 Failover/failback terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544
35-4 TPC for Replication - Metro Mirror copy sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546
35-5 TPC for Replication - Global Mirror copy sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
35-6 TPC for Replication - session concept. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548
35-7 TPC for Replication - four sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
35-8 TPC for Replication system overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
35-9 Replication Manager server connectivity to DS6000 and DS8000 . . . . . . . . . . . . . 554
35-10 Select Configure network ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555
35-11 TPC for Replication server freeze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
35-12 LSS heartbeat triggers freeze when connectivity to server fails . . . . . . . . . . . . . . . 559
35-13 Windows 2003 Software level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560
35-14 GUI URL determined during Replication Manager install time . . . . . . . . . . . . . . . . 561
35-15 TPC for Replication GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562
35-16 Launch the Replication Manager GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563
35-17 Health Overview panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564
35-18 Sessions overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565
35-19 About to create copy sets to session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
35-20 Actions against a session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
35-21 GUI basic layout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
35-22 Select storage subsystem and select View/Modify Details action . . . . . . . . . . . . . . 568
35-23 Storage subsystem details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
35-24 Path overview panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569
35-25 Path overview of a DS8000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
35-26 Replication Management Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
35-27 Advanced tools option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
35-28 Console log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
36-1 Sample update sequence in a database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577
36-2 Sample Rolling Disaster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578
36-3 Dependent write protection of database integrity. . . . . . . . . . . . . . . . . . . . . . . . . . . 579
36-4 eRCMF protection tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
36-5 eRCMF architecture: 2-site configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581
36-6 eRCMF architecture: 3-site configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581
36-7 eRCMF issues freeze command in a disaster recovery scenario . . . . . . . . . . . . . . 582
37-1 TotalStorage Continuous Availability for Windows protection tier . . . . . . . . . . . . . . 588
37-2 Architecture of the solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589
A-1 lspv output before recreating the volume group. . . . . . . . . . . . . . . . . . . . . . . . . . . . 595
A-2 lspv output after recreating the volume group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595
A-3 Target file system stanza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596
xxii IBM System Storage DS8000 Series: Copy Services in Open Environments
A-4 Remote Mirror and Copy phantom disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597
A-5 Original time stamp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597
A-6 Update source time stamp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598
A-7 Update secondary server ODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598
A-8 Microsoft VSS architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
A-9 Microsoft VSS with DS8000 FlashCopy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604
A-10 Microsoft VDS architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605
A-11 QLogic SANsurfer VDS Manager startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607
A-12 Microsoft VDS and VSS architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
A-13 Microsoft VDS and VSS with DS8000 storage subsystem . . . . . . . . . . . . . . . . . . . 609
A-14 Virtual machine source disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
A-15 Virtual disk types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
A-16 FlashCopy within the same virtual machine. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 619
A-17 FlashCopy within the same virtual machine - After FlashCopy . . . . . . . . . . . . . . . . 620
A-18 FlashCopy between virtual machines - Source virtual machine . . . . . . . . . . . . . . . 621
A-19 FlashCopy between virtual machines - Target virtual machine . . . . . . . . . . . . . . . . 622
A-20 FlashCopy between virtual machines - Target virtual machine after FlashCopy . . . 623
A-21 FlashCopy between ESX Servers - Source virtual machine . . . . . . . . . . . . . . . . . . 624
A-22 FlashCopy between ESX Servers - Target virtual machine. . . . . . . . . . . . . . . . . . . 625
A-23 FlashCopy between ESX Servers - Target virtual machine after FlashCopy . . . . . 626
B-1 System i5 partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
B-2 Single level storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632
B-3 Input Output Processors (IOPs). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
B-4 Elements of System i5 Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
B-5 IASPs in System i5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
B-6 Metro mirror of IASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 638
B-7 Functioning of the System i Copy Services Toolkit . . . . . . . . . . . . . . . . . . . . . . . . . 642
B-8 Checking status of IASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
B-9 wrkcfgsts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644
B-10 Status of IASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644
B-11 dspessdta. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645
B-12 IASP name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646
B-13 Display Toolkit options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646
B-14 chkpprc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647
B-15 Checking PPRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 648
B-16 chkpprc completed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
B-17 viewlog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 650
B-18 Toolkit log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
B-19 Viewprof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
B-20 DS CLI profiles on i5/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652
B-21 Display DS CLI profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652
B-22 Select DS CLI script on i5/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653
B-23 DS CLI script on i5/OS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654
B-24 swpprc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654
B-25 Parameter scheduled. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655
B-26 Messages about progress-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656
B-27 Log of SWPPRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657
B-28 Metro Mirror copy if IASP after switchover. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657
B-29 Original IASP after switchover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
B-30 dspessdta after switchover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
B-31 swpprcback to production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 659
B-32 Toolkit log after successful switchback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 660
B-33 Status of IASP on production after switchback . . . . . . . . . . . . . . . . . . . . . . . . . . . . 661
Figures xxiii
B-34 Toolkit status after switchback to production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 661
B-35 swpprc*unscheduled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 662
B-36 Message Waiting for reply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 663
B-37 Display operator message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 663
B-38 Confirming Unscheduled switch by replying the message. . . . . . . . . . . . . . . . . . . . 664
B-39 dspessdta after unscheduled switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665
B-40 SWPPRC *COMPLETE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665
B-41 Waiting for reply to operator message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 666
B-42 Confirm replicate from remote site to local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . 666
B-43 Status after switch is completed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667
B-44 Global Mirror of IASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669
B-45 falovrgmir . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 671
B-46 Metro Mirror of entire disk space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673
B-47 Tagged loadsource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676
B-48 External loadsource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676
B-49 Properties of loadsource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677
B-50 IOP for Boot from SAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677
B-51 LSPPRC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 678
B-52 Script for suspend Metro mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 678
B-53 Performing script for suspend Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 679
B-54 Script for failover of Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 679
B-55 Performing script for Failover of Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 679
B-56 Status of primary volumes after suspend and failover . . . . . . . . . . . . . . . . . . . . . . . 679
B-57 Status of secondary volumes after suspend and failover . . . . . . . . . . . . . . . . . . . . 680
B-58 IPL partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 680
B-59 Script for failback of Metro Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681
B-60 Performing script for failback of Metro Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681
B-61 Pause Metro Mirror or secondary DS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681
B-62 Fail over to the primary DS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681
B-63 Failback on primary DS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 682
B-64 Global Mirror of entire disk space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 683
B-65 Pause GM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686
B-66 Pause Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686
B-67 Failover Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686
B-68 Failback Global Copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687
B-69 Fail over to local site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687
B-70 Failback Global Copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687
B-71 Resume Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687
B-72 Failover Global Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 688
B-73 LsFlash at GM unplanned outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689
B-74 Reverse FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689
B-75 FlashCopy of an IASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 690
B-76 PRPIASPBCK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693
B-77 Executing the command rsmiaspbck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694
B-78 FlashCopy of entire System i5 disk space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696
37-3 Apply Activation Codes panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 712
E-1 ESS Copy Services GUI tasks panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 717
E-2 ESS task information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 717
E-3 ESS GUI FlashCopy window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 718
xxiv IBM System Storage DS8000 Series: Copy Services in Open Environments
Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information about the products and services currently available in your area.
Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM
product, program, or service may be used. Any functionally equivalent product, program, or service that does
not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not give you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.
The following paragraph does not apply to the United Kingdom or any other country where such provisions
are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS
PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and changes in the product(s) and the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without
incurring any obligation to you.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrates programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and
distribute these sample programs in any form without payment to IBM for the purposes of developing, using,
marketing, or distributing application programs conforming to IBM's application programming interfaces.
© Copyright IBM Corp. 2005, 2006. All rights reserved. xxv

Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
Redbooks (logo) ™ DS8000™ POWER4™
eServer™ Enterprise Storage Server® POWER5™
ibm.com® ESCON® Redbooks™
iSeries™ FlashCopy® S/390®
i5/OS® FICON® System i™
z/OS® Geographically Dispersed Parallel System i5™
zSeries® Sysplex™ System p™
AIX 5L™ GDPS® System z™
AIX® HACMP™ System Storage™
AS/400® IBM® System Storage DS™
DB2® NetView® Tivoli®
DFSMSdss™ OS/400® TotalStorage®
DS4000™ Parallel Sysplex® WebSphere®
DS6000™ PowerPC®
The following terms are trademarks of other companies:
SAP, and SAP logos are trademarks or registered trademarks of SAP AG in Germany and in several other
countries.
Java, JDK, Solaris, StorageTek, Sun, and all Java-based trademarks are trademarks of Sun Microsystems,
Inc. in the United States, other countries, or both.
Internet Explorer, Microsoft, Windows NT, Windows Server, Windows, and the Windows logo are trademarks
of Microsoft Corporation in the United States, other countries, or both.
Intel, Pentium, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of
Intel Corporation or its subsidiaries in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, and service names may be trademarks or service marks of others.
xxvi IBM System Storage DS8000 Series: Copy Services in Open Environments
Preface
This IBM® Redbook will help you install, tailor, and configure Copy Services for open systems
environments on the IBM System Storage™ DS8000™. It should be read in conjunction with
The IBM TotalStorage DS8000 Series: Implementation, SG24-6786.
This book gives the details necessary to configure each of the Copy Services functions in an
open systems environment. There is a companion book that supports the configuration of the
Copy Services functions in a z/OS® environment, The IBM TotalStorage DS8000 Series:
Copy Services with IBM Eserver zSeries, SG24-6787.
This book will help you design/create a new Copy Services installation or migrate from an
existing installation, as well as address design functions and changes in terminology from
other IBM Copy Services products. We include hints and tips to maximize the effective
installation.
The team that wrote this redbook

This IBM Redbook was produced by a team of specialists from around the world working for
the International Technical Support Organization, San Jose Center in IBM Mainz, Germany.
Bert Dufrasne is a Certified Consulting IT Specialist and Project Leader for IBM
TotalStorage® products at the International Technical Support Organization, San Jose
Center. He has worked at IBM in various IT areas. Before joining the ITSO, he worked for IBM
Global Services as an Application Architect. He holds a degree in Electrical Engineering.
Gustavo Castets is a Certified IBM IT Specialist and Project Leader working for the IBM
International Technical Support Organization, San Jose Center. While in San Jose, from
2001 to 2004, Gustavo co-authored 12 IBM Redbooks™ and taught IBM classes worldwide in
the area of disk storage systems. Before joining the ITSO, Gustavo was based in Buenos
Aires where he worked in different technical sales support positions for more than 22 years.
Today, in addition to his work with the ITSO, Gustavo works for IBM Global Delivery in
Argentina, as a Storage Specialist supporting accounts from the United States and Europe.
Stephen Baird is an IT Specialist with IBM Global Services. He joined IBM in 1999, working
in open systems server performance management and capacity planning. Since 2002, he has
worked in Storage Area Network and disk storage subsystem support and has gained
experience with Brocade, Cisco, and McData Fibre Channel switches and directors as well as
DS8000, DS4000™, and ESS series hardware. He holds a degree in Mechanical
Engineering from the Massachusetts Institute of Technology, in Cambridge, MA.
Werner Bauer is a certified Consulting IT Specialist in Germany. He has 26 years of

experience in storage software and hardware, as well with S/390® and z/OS. His areas of
expertise include disaster recovery solutions in enterprises utilizing the unique capabilities
and features of the IBM disk storage servers, ESS and DS6000/DS8000. He has written
extensively in various IBM Redbooks on topics such as Transactional VSAM and
DS6000/DS8000 Architecture, Concepts and Copy Services. He holds a degree in
Economics from the University of Heidelberg and in Mechanical Engineering from FH
Heilbronn.
Denise Brown is a Software Engineer at the IBM Open Systems Level Test Lab in Tucson,
Arizona. She has been working with IBM Storage for the past four years, with experience in
© Copyright IBM Corp. 2005, 2006. All rights reserved. xxvii

storage software and hardware in an open system environment. Her current areas of focus
include Copy Services solutions in Metro/Global Mirror and Incremental Re-synchronization
for the DS8000. She holds a degree in Engineering Mathematics.
Jana Jamsek is an IT Specialist in IBM Slovenia. She works in Storage Advanced Technical
Support for Europe as a specialist for IBM Storage Systems and i5/OS® systems. Jana has
eight years of experience in the System i™ and AS/400® area, and six years experience in
Storage. She holds a Master’s degree in computer science and a degree in mathematics from
University of Ljubljana, Slovenia. She has authored serveral IBM Redbooks and IBM
Redpapers on iSeries™ and TotalStorage.
Wenzel Kalabza is an IT Specialist in IBM Germany. He started in 1998 as a field quality

engineer for IBM hard disk drives and was technical lead in HDD robustness and rotational
vibration testing. He has three years of experience supporting IBM storage products. As a
member of the DASD EMEA back office, he initially supported the ESS. and is now a Product
Field Engineer for the DS6000™, specializing in Copy Services.
Peter Klee is an IT Specialist with ATS EMEA, located in Mainz, Germany. He has 10 years
of experience in data centers. He worked for a large bank in Germany and was responsible
for the architecture and the implementation of the disk storage environment using EMC
Symmetrix, HDS Lightning, and ESS Model 800. He joined IBM in 2003, where he was
working for Strategic Outsourcing. Since June 2004 he is working in the ATS System Storage
team in Mainz. His main focus is Copy Services in the open systems environment with
DS8000 and DS6000, especially Global Mirror and Metro/Global Mirror.
Markus Oscheka is an IT Specialist for Proof of Concepts and Benchmarks at the ATS
Customer Solutions team in Mainz, Germany. His areas of expertise include setup and
demonstration of IBM System Storage products and solutions in various environments
including AIX®, Linux®, Windows®, HP-UX, and Solaris™. He has worked at IBM for five
years. He has performed many Proof of Concepts with Copy Services on DS6000/DS8000,
as well as Performance-Benchmarks with DS4000/DS6000/DS8000.
Ying Thia is an Advisory IT Specialist based in IBM Singapore, providing storage technical
support. She has 14 years of experience in the S/390 and storage environment. Before
joining the IBM Singapore Storage team, she worked in IBM Global Services where her
responsibilities included technical support and services delivery. Her areas of expertise
include IBM high-end disk and tape storage subsystems and disaster recovery solutions
using the capabilities and features of IBM storage products. She co-authored a previous IBM
Redbook and workshop for zSeries® Copy Services.
Robert Tondini is a Senior IT Specialist based in IBM Australia, providing storage technical
support. He has 12 years of experience in the S/390 and storage environment. His areas of
expertise include IBM high-end disk and tape storage subsystems and disaster recovery
solutions using the capabilities and features of IBM storage products. He co-authored several
IBM Redbooks and a workshop for zSeries copy services.
xxviii IBM System Storage DS8000 Series: Copy Services in Open Environments
The team: Gustavo, Robert, Wenzel, Jana, Peter, Markus, Denise, Werner, Ying, Stephen, Bertrand
Special thanks to
John Bynum and Robert Moon - Techincal Support Marketing Leads
IBM US
We want to thank Michael Eggloff and Peter Klee for hosting us at the European Storage
Competency Center in Mainz, Germany. They were able to supply us with the needed
hardware, conference room, and all of the many things needed to run a successful residency.
Günter Schmitt, Uwe Schweikhard, Edgar Strubel (ATS - IBM Mainz) for their help in
reserving and preparing the equipment we used.
Monika Baier, Susanne Balzer, Marion Barlen, Marion Hartmann, Andrea Witkowski, Gertrud
Kramer - IBM Mainz, for their help with administrative tasks.
Many thanks to the authors of the previous edition of this IBM Redbook:
Peter Kimmel, Jukka Myyrlainen, Lu Nguyen, Gero Schmidt, Shin Takata, Anthony
Vandewerdt, Bjoern Wesselbaum
We also would like to thank:
Selwyn Dickey, Timothy Klubertanz, Vess Natchev, James McCord and Chuck Stupca
IBM Rochester - System i Client Technology Center
Guido Ihlein, Marcus Dupuis, Wilfried Kleemann

IBM Germany
Bob Bartfai, Jennifer Mason, James Davison, Steve Van Gundy, Richard Ripberger, Bill
Raywinkle, Christopher o’Toole, Jim Sedgwick, Henry Sautter, Craig Gordon, Rosemary
McCutchen,, Lee La Frese, Alan McClure, Rachel Mikolajewski, Gail Spear, Leann Vaterlaus,
David V Valverde, Sonny Williams
Brian Sherman
IBM Canada
Nick Clayton
IBM UK
Preface xxix
Many thanks to Emma Jacobs and Julie Czubik.
Become a published author

Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with
specific products or solutions, while getting hands-on experience with leading-edge
technologies. You'll team with IBM technical professionals, Business Partners and
customers.
Your efforts will help increase product acceptance and customer satisfaction. As a bonus,
you'll develop a network of contacts in IBM development labs, and increase your productivity
and marketability.
Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html
Comments welcome
Your comments are important to us!
We want our Redbooks to be as helpful as possible. Send us your comments about this or
other Redbooks in one of the following ways:
򐂰 Use the online Contact us review IBM Redbook form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
redbook@us.ibm.com
򐂰 Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400
xxx IBM System Storage DS8000 Series: Copy Services in Open Environments
Summary of changes
This section describes the technical changes made in this edition of the book and in previous
editions. This edition may also include minor corrections and editorial changes that are not
identified.
Summary of Changes
for SG24-6788-02
for IBM System Storage DS8000 Series: Copy Services in Open Environments
as created or updated on November 28, 2006.
November 2006, Third Edition

This revision reflects the addition, deletion, or modification of new and changed information
described below.
New information
򐂰 3-site Metro/Global Mirror with Incremental Resync
򐂰 IBM TotalStorage Productivity Center for Replication
򐂰 Host operating systems considerations for System i
򐂰 New DS8000 series Turbo models: 93x/9B2
Changed information
򐂰 Updated Copy Services examples
򐂰 Updated licensing information
© Copyright IBM Corp. 2005, 2006. All rights reserved. xxxi

xxxii IBM System Storage DS8000 Series: Copy Services in Open Environments
Part 1
Part 1 Overview
In this part we describe the various Advanced Copy Services offerings for the DS8000 series
and how they relate to previous Copy Services offerings available on the Enterprise Storage
Server® (ESS).
This part will also show how the existing Copy Services functions from the ESS can coexist
with the DS8000 series Copy Services. Similarly, we discuss their use with the DS6000
series Copy Services.
With the announcement of the DS8000 series and Advanced Copy Services for z/OS on the
DS8000 series, we introduced some new terminology. We also introduced these terms for
Advanced Copy Services Version 2 for the ESS. These are detailed in the following IBM
Redbook, The IBM TotalStorage DS8000 Series: Implementation, SG24-6786, and are
summarized in Table 1.
Table 1 Reference chart for DS Copy Services on DS8000

DS8000 function ESS800 Version 2 function Formerly known as
FlashCopy® FlashCopy FlashCopy
Global Mirror Global Mirror Asynchronous PPRC
Metro Mirror Metro Mirror Synchronous PPRC
Global Copy Global Copy PPRC Extended Distance
z/OS Global Mirror z/OS Global Mirror Extended Remote Copy (XRC)
Metro/Global Mirror for zSeries z/OS Metro/Global Mirror 3-site solution using Sync
PPRC and XRC
Metro/Global Copy Metro/Global Copy 2 or 3 site Asynchronous

Cascading PPRC
© Copyright IBM Corp. 2005, 2006. All rights reserved. 1

The Copy Services configuration is done using either the IBM System Storage DS™ Storage
Command-Line Interface (DS CLI) or the IBM System Storage DS Storage Manager Copy
Services Graphical User Interface (DS GUI).
It should be noted:
򐂰 All DS8000 installations require at least an Operating Equipment License (OEL) key to
operate. However, Copy Services functions can only be used where the appropriate
License Key is installed.
򐂰 The key Copy Services license types are Operating Equipment License (OEL), Remote
Mirror and Copy or PPRC (RMC), and Point-in-Time Copy or FlashCopy (PTC). For
DS8000 there is also Remote mirror for z/OS (RMZ).
򐂰 The DS CLI replaces both the ESS CLI and the ESS Copy Services CLI.
򐂰 The DS CLI can also be used for ESS 800 Copy Services, but not ESS configuration. For
ESS configuration you need to continue to use the ESS Web Specialist or ESS CLI.
򐂰 The DS CLI can invoke Remote Mirror relationships with the ESS 800, if the ESS 800 is at
code level LIC 2.4.3.65 or later.
򐂰 The DS CLI can be used in reusable scripts, and provides a consistent interface for
current and planned IBM System Storage products.
򐂰 The DS CLI now invokes a Copy Services function directly rather than invoking a saved
task.
򐂰 The DS GUI can only be used for one-time execution of a Copy Services operation; it
cannot save tasks.
2 IBM System Storage DS8000 Series: Copy Services in Open Environments

1
Chapter 1. Introduction
This chapter provides a brief summary of the various Copy Services functions available on
the DS8000 series. These services are very similar to the existing Copy Services for the IBM
Enterprise Storage Server, and some models of the ESS are interoperable with the DS8000
as well as the DS6000.

1.1 Introduction to Copy Services
We briefly describe each function of Copy Services in this section. Copy Services are a
collection of functions that provide for disaster recovery, data migration, and data duplication
functions. There are two primary types of Copy Services functions: Point-in-Time Copy and
Remote Mirror and Copy. Generally, the Point-in-Time Copy function is used for data
duplication and the Remote Mirror and Copy functions are used for data migration and
disaster recovery.
With the Copy Services functions, for example, you can create backup data with little or no
disruption to your application, and you can back up your application data to the remote site for
disaster recovery.
Copy Services run on the DS8000 Storage Unit and support open systems and zSeries
environments. These functions are supported also on the previous generation of storage
systems, the IBM TotalStorage Enterprise Storage Server (ESS).
Many design characteristics of the DS8000 and the data copying and mirroring capabilities of
the Copy Services features contribute to the protection of your data, 24 hours a day and 7
days a week (24x7). The licensed features included in Copy Services are as follows:
򐂰 FlashCopy, which is a Point-in-Time Copy function
򐂰 Remote Mirror and Copy functions, previously known as Peer-to-Peer Remote Copy or
PPRC, which include:
– Metro Mirror, previously known as Synchronous PPRC
– Global Copy, previously known as PPRC Extended Distance
– Global Mirror, previously known as Asynchronous PPRC
– 3-site Metro/Global Mirror with Incremental Resync
򐂰 z/OS Global Mirror, previously known as Extended Remote Copy (XRC)
򐂰 z/OS Metro/Global Mirror
We explain these functions in more detail in the next section.
You can manage the Copy Services functions through a command-line interface (DS CLI)
and a new Web-based interface (DS Storage Manager). You also can manage the Copy
Services functions through the open application programming interface (DS Open API).
When you manage the Copy Services through these interfaces, these interfaces invoke Copy
Services functions via the Ethernet network. In zSeries environments, you can invoke the
Copy Service functions by TSO commands, ICKDSF, the DFSMSdss™ utility, and so on.
We explain these interfaces in Part 2, “Interfaces” on page 15.
1.1.1 Point-in-Time Copy (FlashCopy)

The Point-in-Time Copy feature, which includes FlashCopy, enables you to create full volume
copies of data in a Storage Unit. To use it, you must purchase the Point-In-Time Copy 2244
function authorization model, which is 2244 Model PTC.
When you set up a FlashCopy operation, a relationship is established between the source
and target volumes, and a bitmap of the source volume is created. Once this relationship and
bitmap are created, the target volume can be accessed as though all the data had been
physically copied. While a relationship between the source and target volume exists,
optionally, a background process copies the tracks from the source to the target volume.

When a FlashCopy operation is invoked, it takes only a few seconds to complete the process
of establishing the FlashCopy pair and creating the necessary control bitmaps. Thereafter,
you have access to a Point-in-Time Copy of the source volume. As soon as the pair has been
established, you can read and write to both the source and target volumes.
After creating the bitmap, a background process begins to copy the real data from the source
to the target volumes. If you access the source or the target volumes during the background
copy, FlashCopy manages these I/O requests, and facilitates both reading from and writing to
both the source and target copies. When all the data has been copied to the target, the
FlashCopy relationship is ended.
1.1.2 Remote Mirror and Copy RMC (formerly Peer-to-Peer Remote Copy)
The new Remote Mirror and Copy license authorization enables several Copy Services
functions dependent on the configuration and settings used.
The Remote Mirror and Copy feature (RMC) (formerly called Peer-to-Peer Remote Copy, or
PPRC) is a flexible data mirroring technology that allows replication between volumes on two
or more disk storage systems. You can also use this feature for data backup and disaster
recovery. Remote Mirror and Copy is an optional function. To use it, you must purchase the
Remote Mirror and Copy 2244 function authorization model, which is 2244 Model RMC.
Attention: For the DS8000 Turbo models 931, 932, and 9B2, the Metro Mirror and Global
Mirror features are licensed separetely.
DS8000 Storage Units can participate in Remote Mirror and Copy solutions with the ESS
Model 750, ESS Model 800, and DS6000 Storage Units. To establish an RMC (formerly
PPRC) relationship between the DS8000 and the ESS, the ESS needs to have licensed
internal code (LIC) Version 2.4.3.15 or later.
The Remote Mirror and Copy feature can operate in the following modes.
Metro Mirror (formerly Synchronous PPRC)

Metro Mirror provides real-time mirroring of logical volumes between two DS8000s that can
be located up to 300 km from each other. It is a synchronous copy solution where write
operations are completed on both copies (local and remote site) before they are considered
to be complete.
Global Copy (formerly PPRC-XD)

Global Copy copies data non-synchronously and over longer distances than is possible with
Metro Mirror. When operating in Global Copy mode, the source volume sends a periodic,
incremental copy of updated tracks to the target volume, instead of sending a constant
stream of updates. This causes less impact to application writes for source volumes and less
demand for bandwidth resources, while allowing a more flexible use of the available
bandwidth.
Global Copy does not keep the sequence of write operations. Therefore, the copy is normally
fuzzy, but you can make a consistent copy through synchronization (called a go-to-sync
operation). After the synchronization, you can issue FlashCopy at the secondary site to make
the backup copy with data consistency. After the establishment of the FlashCopy, you can
change the PPRC mode back to the non-synchronous mode.
Chapter 1. Introduction 5
Note: When you change PPRC mode from synchronous to non-synchronous mode, you
change the PPRC mode from synchronous to suspend mode at first, and then you change
PPRC mode from suspend to non-synchronous mode.
If you want make a consistent copy with FlashCopy, you must purchase a Point-in-Time Copy
function authorization (2244 Model PTC) for the secondary Storage Unit.
Global Mirror (Asynchronous PPRC)

Global Mirror provides a long-distance remote copy feature across two sites using
asynchronous technology. This solution is based on the existing Global Copy and FlashCopy.
With Global Mirror, the data that the host writes to the Storage Unit at the local site is
asynchronously shadowed to the Storage Unit at the remote site. A consistent copy of the
data is automatically maintained on the Storage Unit at the remote site.
Global Mirror operations provide the benefit of supporting operations over virtually unlimited
distances between the local and remote sites, restricted only by the capabilities of the
network and the channel extension technology. It can also provide a consistent and
restartable copy of the data at the remote site, created with minimal impact to applications at
the local site.
The ability to maintain an efficient synchronization of the local and remote sites with support
for failover and failback modes helps to reduce the time that is required to switch back to the
local site after a planned or unplanned outage.
3-site Metro/Global Mirror with Incremental Resync

Metro/Global Mirror combines a Metro Mirror and a Global Mirror together to provide the
possibility to implement a 3-site disaster recovery solution. The production is using the
storage at the local site, which is replicated synchronously using a Metro Mirror to an
intermediate site. The secondary volumes of the Metro Mirror are further on used as the
primary volumes to the cascaded Global Mirror, which replicates the data to the remote
disaster recovery site.
This provides a very resilient and flexible solution to recover in various disaster situations.
Customers also benefit from a synchronous replication of the data to a close location acting
as the intermediate site. It also enables the possibility to copy the data across almost
unlimited distance, whereby data consistency can be provided in any time in each location.
With Incremental Resync it is possible to change the copy target destination of a copy relation
without requiring a full copy of the data. This functionality can be used, for example, when an
intermediate site fails due to a disaster. In this case a Global Mirror will be established from
the local to the remote site, which bypasses the intermediate site. When the intermediate site
becomes available again the Incremental Resync is used to bring it back into the
Metro/Global Mirror setup.
The 3-site Metro/Global Mirror is an optional chargeable feature available on all models
(92x/9Ax and 93x/9Bx). It requires Licensed Machine Code (LMC) level 5.2.200.x (bundle
version 6.2.200.x) or later.
Additional licensing information for Metro/Global Mirror can be found in Appendix D,

“Licensing” on page 709.

2
Chapter 2. Copy Services architecture

This chapter is an overview of the structure of the Copy Services communication architecture
in either an open environment or a zSeries environment. The chapter covers the following
topics:
򐂰 Introduction to the Coppy Services structure
򐂰 How the new structure of Copy Services works
򐂰 System z communication path for Copy Services

2.1 Introduction to the Coppy Services structure
The Copy Services architecture for the IBM ESS 800 used a construct called a Copy Services
Domain to manage Copy Services. The communications architecture for the DS6000 and
DS8000 is noticeably different. Instead of a Copy Services Domain, we now use the concept
of Storage Complexes.
You can perform Copy Services operations both within or between Storage Complexes. An
ESS 800 Copy Services Domain (at the correct firmware level) can, for management
purposes, appear to a DS6000 or DS8000 as an independent Storage Complex.
Now within a Storage Complex you will find management consoles, Storage Units, and in the
case of the DS8000, Storage Facility Images. First we define each of these constructs.
2.1.1 Management console defined

A management console is a PC that is used to manage the devices within a Storage
Complex. In the case of the DS6000, it is a PC with a Windows operating system, onto which
the user has installed the DS Storage Manager Console software. This is usually referred to
as a Storage Management Console or SMC. In the case of the DS8000, it is a dedicated PC
that comes with a pre-installed Linux-based operating system and pre-installed management
software. The DS8000 console is called a Hardware Management Console (HMC).
In either case, the end-user interacts with the management console using either the Web
browser based DS GUI or the Command-Line Interface (DS CLI). It is possible using the DS
GUI to manage Copy Services operations on multiple Storage Complexes.
2.1.2 Storage Unit defined

A Storage Unit is the physical storage device (including expansion enclosures) that you see
when you walk into the computer room. If you open a rack door and look at a DS6000 system
enclosure (a 1750-511 or 522) and all its attached expansion enclosures (1750-EX1 or EX2),
then you are looking at a single DS6000 Storage Unit. If you then look at a DS8000 sitting on
the floor (with any attached expansion racks), then you are looking at a DS8000 Storage Unit.
An example would be a 2107-922 or 932 with an attached 2107-9E2. Another example would
be a 2107-9A2 with an attached 2107-9AE. In each example you would have a single
DS8000 Storage Unit.
2.1.3 Storage Facility Image (SFI) defined

The Storage Facility Image construct is different for the DS6000 and DS8000.

DS6000 SFI
For the DS6000, the SFI concept does not really apply. You could say that each DS6000
Storage Unit contains one SFI. However, in general a DS6000 is always referred to as simply
a Storage Unit. When using the DS CLI, there are separate commands to display a DS6000
Storage Unit or a DS6000 SFI. However, both will refer to the same device. Using the DS GUI,
you only work with the DS6000 Storage Unit. The GUI will not offer an option to select a
DS6000 SFI. In Example 2-1 we connect to a DS6000 Management Console using the DS
CLI. We issue the lssu command to display the DS6000 Storage Unit and the lssi command
to display the DS6000 Storage Facility Image. In each case we see the same serial number
and the same World Wide Node name (WWNN). The Storage Unit and the SFI are the same.
Example 2-1 Difference between a DS6000 Storage Unit and DS6000 SFI
dscli> lssu
Date/Time: 13 November 2005 19:21:17 IBM DSCLI Version: 5.1.0.204
Name ID Model WWNN pw state
=========================================================
13-00247 IBM.1750-1300247 511 500507630EFFFE16 On
dscli> lssi
Name ID Storage Unit Model WWNN State ESSNet
============================================================================
- IBM.1750-1300247 IBM.1750-1300247 511 500507630EFFFE16 Online Enabled
DS8000 SFI
For the DS8000, the SFI concept is necessary because a DS8000 can be ordered as a model
9A2 or 9B2. In these models all resources in the DS8000 are divided between two separate
machine images. This means that to all attached hosts, a single DS8000 Storage Unit
effectively contains two DS8000s. It is like having two separate machines in one physical box.
This means that when using the DS GUI or the DS CLI, you have to be very clear whether
you are working with the SFI or with the Storage Unit. For most operations you always work
with an SFI. You can tell the difference very easily. The serial number of the Storage Unit
always ends in zero. The serial number of the first SFI always ends in 1. If you have ordered
a model 2107-9A2 or 2107-9B2 then you will also have a second SFI. The serial number of
the second SFI always ends in 2.
DS8000 Model 921 and 922 SFI example

In Example 2-2 we connect to a DS8000 Management Console using the DS CLI. We issue
the lssu command to display the DS8000 Storage Unit of a 2107-922. We then issue the lssi
command to display the DS8000 Storage Image on that Storage Unit. The Storage Unit serial
number is 7520780. The SFI serial number is 7520781. Note how the Storage Unit and the
SFI have different WWNNs.
Example 2-2 Difference between a DS8000 Storage Unit and DS8000 SFI - Model 922
dscli> lssu
=============================================================
2107-7520780 IBM.2107-7520780 922 5005076303FFF9A5 On
dscli> lssi
=================================================================================
ATS_3_EXP IBM.2107-7520781 IBM.2107-7520780 922 5005076303FFC1A5 Online Enabled
Chapter 2. Copy Services architecture 9

DS8000 Model 9A2 SFI example
In Example 2-3 we connect to a DS8000 Management Console using the DS CLI. We issue
the lssu command to display the DS8000 Storage Unit of a 2107-9A2. We then issue the
lssi command to display the DS8000 Storage Images on that Storage Unit. The Storage Unit
serial number is 7520780. The SFI serial numbers are 7520781 and 7520782. The Storage
Unit and the SFIs all have different WWNNs.
Example 2-3 Difference between a DS8000 Storage Unit and DS8000 SFI - Model 9A2
dscli> lssu
================================================================
2107_75ABTV1/V2 IBM.2107-75ABTV0 9A2 5005076303FFFE63 On
dscli> lssi
============================================================================
- IBM.2107-75ABTV1 IBM.2107-75ABTV0 9A2 5005076303FFC663 Online Enabled
- IBM.2107-75ABTV2 IBM.2107-75ABTV0 9A2 5005076303FFCE63 Online Enabled
2.1.4 What is a Storage Complex

A Storage Complex can be one or two DS8000 Storage Units managed by one or two DS
Hardware Management Consoles (HMCs). A Storage Complex could also be one or two
DS6000 Storage Units, managed by one or two DS Storage Management Consoles (SMCs).
In Figure 2-1, you see a logical view of two Storage Complexes, each with one DS8000
Storage Unit. They are running Remote Mirror and Copy. Now because the two DS HMCs are
linked via Ethernet, you could connect using the DS GUI to either HMC and manage Copy
Services on both DS8000s.
Remote Mirror Copy Links
DS8000 DS8000
DS HMC DS HMC
Ethernet Link
Storage Storage
Complex 1 Complex 2
Figure 2-1 Logical view of an Storage Complex

Currently for the DS8000 and DS6000, we can manage two Storage Units in one Storage
Complex. For ESS 800, we can manage up to eight ESS 800s in a single ESS Copy Services
domain.
Note: Only machines of the same type, like 2107, can be in the same Storage Complex.
However, Storage Complexes with different machine types can be joined together. For
example, a DS8000 Storage Complex and a DS6000 Storage Complex can be
inter-connected.
2.2 How the new structure of Copy Services works

With the exception of System z™ commands (see 2.3, “System z communication path for
Copy Services” on page 14), communication to a DS6000 or DS8000 needs an available DS
SMC or DS HMC. See Figure 2-2.
Client Client Client

DS CLI or DS CLI or DS CLI or
Web browser Web browser Web browser
DS HMC DS SMC
Network Interface Network Interface
Server Server
SFI
Network Network Network Network
Copy
Interface Interface Interface Interface
Services
Node Node Node Node
Server
Microcode Microcode
Microcode
Server 0 Server 1 Processor Processor

Controller 0 Controller 1
complex1 complex2
DS8000 DS6000 ESS800

Figure 2-2 Network Interface communication structure
DS8000 management structure

Using Figure 2-2 we can see that:
򐂰 The client uses either the DS CLI or a Web browser GUI to issue commands to the
Network Interface Server running on the DS HMC.
򐂰 The Network Interface Server software that resides on the HMC will communicate with the
Client.

򐂰 The Network Interface Server software will communicate with the Network Interface Node,
which resides on each server of a Storage Facility Image. From this point, the Network
Interface will then talk to the microcode, which operates the DS8000.
1750 DS6000 management structure

Using Figure 2-2 on page 11 we can see that:
򐂰 The client uses either the DS CLI or a Web browser GUI to issue commands to the
Network Interface Server running on the DS SMC.
򐂰 The DS Network Interface Server software will reside on the DS SMC. This is what the
client will communicate with.
򐂰 The Network Interface Server software will communicate with the Network Interface Node,
which resides on each controller of the DS6000. From this point the Network Interface will
then interact with the microcode that operates the controllers.
2105 ESS 800 management structure

Using Figure 2-2 on page 11 we can see that:
򐂰 The client uses either DS CLI or a ESS Copy Services Web browser GUI to issue
commands to the ESS 800 Copy Services Server running on an ESS 800 cluster. The
client could also use the DS GUI to issue commands to the ESS 800 Copy Services
Server if a DS6000 SMC or DS8000 HMC is available to route them through (not shown in
the diagram).
򐂰 The ESS 800 Copy Services Server will then interact with the microcode that operates the
ESS 800.

2.2.1 Remote Mirror and Copy between Storage Complexes
It is possible to use Remote Mirror and Copy between Storage Complexes, as depicted in
Figure 2-3. In this scenario we have three Storage Complexes. The complexes are
interconnected at the top of the chart by Storage Area Network (SAN) connections (solid
lines), and at the bottom of the chart by an Ethernet LAN (dashed lines).
SAN A SAN B
ESS Copy
Services
Domain
(Complex 3)
ESS
ESS
800
800
Storage Storage
DS6000
DS6000
DS8000
DS8000
Complex Complex
1 2
HMC1 SMC1
LAN
Client 1 Client 2
Figure 2-3 Remote Mirror and Copy between Storage Complexes
2.2.2 Differences between the DS CLI and the DS GUI

The DS CLI is not capable of managing several domains in a single session. However, if a
single DS CLI client machine has network access to all Storage Complexes, then that client
could issue concurrent DS CLI commands to each complex. Each complex would be
managed by a separate DS CLI session. The DS CLI can be script driven, and of course
scripts can be saved. So by using DS CLI, you can achieve automation.
The DS GUI is accessed via a Web browser. The DS GUI is not able to save tasks as we do
on the ESS. Thus, you cannot use the DS GUI to initiate a series of saved tasks. To do this we
need to use the DS CLI. If you wish to use a single GUI to manage multiple Storage
Complexes, then you need to define in the GUI all of the Storage Complexes. This will allow
you to manage FlashCopy or Remote Mirror and Copy on, or between, every Storage Unit in
every defined and accessible Storage Complex.
When you look at the structure in Figure 2-2 on page 11, you can see that you need a
working DS SMC or DS HMC in every Storage Complex, to communicate with that complex’s
DS6000 or DS8000. For an inter-Storage Complex Remote Mirror and Copy, the DS GUI will
establish sessions to the source and target Storage Complexes to show all paths and LUNS.
If no DS HMC or DS SMC is available at the remote Storage Complex, you cannot select this
Storage Complex as a target.
It is possible to have two SMCs or two HMCs in a Storage Complex for the purpose of
redundancy.

2.3 System z communication path for Copy Services
In a z/OS environment, you can issue Copy Services commands to the DS6000 and DS8000
via inband communication. This is done by sending commands via a FICON® (or ESCON® in
the case of DS8000) host link to a conduit CKD volume. From there it gets passed to the
microcode for execution. This is exactly the same as what is done with an ESS 800.
Depending on the operating system, various software packages are available to issue
commands; examples include TSO or ICKDSF.
The ability to send inband commands does not necessarily mean that the DS CLI or DS GUI
do not have a role to play. To be able to issue a command to a DS6000 or DS8000, a System
z operating system needs to be able to communicate with the relevant Storage Unit. We may
have a remote Storage Unit that is not connected via FICON (or ESCON in the case of a
DS8000) to an active z/OS. This means we would not be able to access a conduit CKD
volume on the remote Storage Unit. Now, provided we have TCP/IP connectivity and an
appropriate management station (such as a Windows host with a Web browser or DS CLI
installed), we could use the DS CLI or the DS GUI to manage the remote Storage Unit. We
could then use inband commands to manage the local Storage Unit. If the remote and local
Storage Units were connected by RMC links, then we can also send some commands inband
via the RMC links, such as commands to establish FlashCopies of the remote RMC target
volumes.

Part 2
Part 2 Interfaces
In this part we discuss the interfaces available to manage the Copy Services features of the
DS8000. We give an overview of the interfaces, discuss the options available, discuss
configuration considerations, and give usage examples of the interfaces.

3
Chapter 3. DS Storage Manager (GUI)

This chapter provides an introduction to the DS Storage Manager, which can be used to
configure and administer the DS storage system. The IBM System Storage DS Storage
Manager is an interface that is used to perform logical configuration and Copy Services
management.
We cover these topics:

򐂰 Connecting to the DS8000 DS GUI
򐂰 Managing the Storage Complex
򐂰 Creating a DS Storage Management Console PC
򐂰 Accessing the Information Center

3.1 Connecting to the DS8000 DS GUI
To connect to the DS8000 DS GUI through a Web browser, enter the Web site (URL) of
Hardware Management Console 1 (HMC1) or, if present, HMC2. The Web site consists of a
TCP/IP address, or a fully qualified name that the DNS server can resolve, with the correct
port number. If, for instance, your HMC IP address is 10.0.0.1 and you connect to
http://10.0.0.1, you will not get a successful connection. This is because the DS Storage
Manager is accessible by connecting to either:
http://10.0.0.1:8451/DS8000
or
https://10.0.0.1:8452/DS8000
Clearly, you should change the IP address to match the one used by your HMC.
Logging in to the DS Storage Manager requires that you provide your user name and
password. This function is generally administered through your system administrator and by
your company policies. The preconfigured user ID and password are as follows:
򐂰 User ID: admin
򐂰 Password: admin
When you log on to the DS GUI for the first time using the admin user ID, you will be forced to
change the password from admin to something more secure.
Accessing the Storage Manager when logged onto the HMC

When logged onto the DS8000 HMC directly, you can also access the DS Storage Manager.
Log on to the HMC with:

򐂰 User ID:customer
򐂰 Password:cust0mer
Then right-click the HMC desktop and select Net → Browser → DS Storage Manager. This
will start the Opera Web Browser. You will not be able to go to any other Web sites from here.
Note: The ability to use the HMC to access the DS GUI means you do not need a separate
PC to configure or manage the DS8000, unless you wish to use the DS CLI. The DS CLI is
not installed on the HMC so it must be installed on an Ethernet attached server or PC.
3.1.1 Advantages of using the DS GUI over the DS CLI

The DS Storage Manager GUI has the following advantages for managing Remote Mirror and
Copy functions:
򐂰 It requires less specialized skills for effective use.
򐂰 It is a real-time graphical interface to the DS8000 subsystem, which may present
information more easily to some users.
򐂰 You do not need to install anything on your PC in order to manage Copy Services — a
simple Web browser pointed to the DS HMC is enough.

3.1.2 Disadvantages of using the DS GUI over the DS CLI
The DS Storage Manager GUI has the following disadvantages for managing Remote Mirror
and Copy functions:
򐂰 You cannot perform all supported functions from this interface.
򐂰 You cannot use this interface for automated actions.
򐂰 You cannot save your actions and reuse them later.
You still need to understand the implications of the actions you are performing (for example,
you need to understand that a FlashCopy will overwrite the target LUN or volume).
3.2 Managing the Storage Complex

A DS8000 is managed by either one or two HMCs. These HMCs and the Storage Units they
manage represent a Storage Complex. In Figure 3-1 we have connected to the Storage
Complex managed by the HMC at the 10.0.0.1 IP address. This complex consists of one
Storage Unit, as indicated by the arrow.
10.0.0.1
Figure 3-1 A single Storage Complex with a single Storage Unit
A simple scenario may be that we have two sites. At the production site we have a single
DS8000 managed by its own HMC. Then at a remote site we have a second DS8000
managed by its own HMC. There is Ethernet connectivity between the two HMCs. Because
each HMC manages only the DS8000 at its own site, this means we have two Storage
Complexes. If we wish to use a single GUI to manage Copy Services at either site, then we
have to add these Storage Complexes to each other. Once we have done this, we can use
the DS GUI connected to one HMC to manage Copy Services on either Storage Unit at either
site, plus set up Copy Services relations between the two sites.
3.2.1 Procedure to add a Storage Complex
Note: To use the DS GUI supplied by the HMC in one Storage Complex to manage the
Storage Units of the other Storage Complex, you have to do this operation once in each
direction. This means you have to add Complex1 to Complex2, and then add Complex2 to
Complex1.
Important: Make sure the user ID you use to log on to the DS8000 DS GUI also exists on
the other DS8000 Storage Complex and that it has the same password. If not, the
operation to add the Storage Complex will fail. You will need to always use this same user
ID for multi-complex management.
Chapter 3. DS Storage Manager (GUI) 19

To add a Storage Complex:
1. Connect to the DS8000 HMC using the DS GUI.
2. Click Real-time manager.
3. Click Manage hardware.
4. Click Storage complexes.
5. Select Add. Figure 3-2 shows the Add Storage Complex panel.
Figure 3-2 Add Storage Complex panel
6. Enter the IP address of the other Storage Complex’s HMC into the Management console 1
IP address box. If the other Storage Complex has a second HMC, select the Define a
second Management console box and enter the second HMC address into the
Management console 2 IP address box. Now select OK.
7. When the add Storage Complex operation completes, the Storage Complex screen will
now show the DS8000 HMC as an additional Storage Complex. In Figure 3-3 we can see
the successful result. We have added a second Storage Complex that manages one
Storage Unit.
10.0.0.1
10.0.0.100
Figure 3-3 Successful addition of a Storage Complex
8. Having added one DS8000 Storage Complex to another, you are now able to use the
DS8000 DS GUI to create paths and Remote Mirror and Copy pairs between either of the
Storage Units. You can also use the DS8000 DS GUI to manage FlashCopy pairs on
either DS8000. This is assuming all the relevant licenses are present.
Note: The steps to add one DS8000 Storage Complex to another DS8000 Storage
Complex cannot be performed using the DS CLI.

3.3 Creating a DS Storage Management Console PC
With the DS8000, the software is effectively pre-installed onto the DS8000 HMC. This means
that there is no requirement to purchase a separate PC to manage the DS8000. However,
there remains the option to actually install the DS Storage Manager software onto a separate
PC. The main reason to do this is that it will allow you to run a Simulated Manager for the
purposes of creating offline storage configurations. These configurations can be applied at a
later time. However, for the purposes of Copy Services, there is no benefit to having a
separate DS Storage Management PC. Consequently, it will not be discussed in this section.
Note that a separate PC or server is still needed if you wish to use the DS CLI.
3.4 Accessing the Information Center

The Software Information Center displays product or application information. The system
provides a graphical user interface for browsing and searching online documentation. The
broad range of topics covered includes accessibility, Copy Services, device storage, host
system attachments, concurrent code loads, input/output configuration programs, and volume
storage.
Accessing the Information Center using a Web browser

You can connect to the Information Center using a Web browser. If your HMC IP address is
10.0.0.1 then you can access the Information Center running on that HMC at:
http://10.0.0.1:8455
If you wish to access the public version of the DS8000 Information Center, it can be located
at:
http://publib.boulder.ibm.com/infocenter/ds8000ic/index.jsp
Accessing the Information Center from the DS GUI

The help panels are accessed by selecting the? symbol on the upper-right side of the screen,
as indicated by the arrow in Figure 3-4.
Figure 3-4 Help option location
Chapter 3. DS Storage Manager (GUI) 21

The help panel is opened automatically on the current subject of the configuration screen, as
shown in Figure 3-5. In this example the FlashCopy main page was open when help was
called, so the FlashCopy help main page was opened.
Figure 3-5 FlashCopy help main page

4
Chapter 4. DS Command-Line Interface

This chapter provides an introduction to the DS Command-Line Interface (DS CLI), which can
be used to configure and to administer the DS storage system. We describe how you can use
the DS CLI to manage Copy Services relationships.
In this chapter we describe:

򐂰 System requirements
򐂰 Command modes
򐂰 The commands
򐂰 User assistance
򐂰 Return codes

4.1 Introduction and functionality
The IBM System Storage DS Command-Line Interface enables open systems hosts to invoke
and manage FlashCopy and Remote Mirror and Copy functions through batch processes and
scripts.
The command-line interface provides a full-function command set that allows you to check
your Storage Unit configuration and perform specific application functions when necessary.
Before you can use the DS CLI commands, you must ensure the following:
򐂰 It must have been installed as a Full Management Console installation management type.
򐂰 Your Storage Unit must be configured (part of the DS Storage Manager post-installation
instructions).
򐂰 You must activate your license activation codes before you can use the DS CLI
commands associated with the Copy Services functions.
The following list highlights a few of the specific types of functions that you can perform with
the DS Command-Line Interface:
򐂰 Create user IDs that can be used with the GUI and the DS CLI.
򐂰 Manage user ID passwords.
򐂰 Install activation keys for licensed features.
򐂰 Manage Storage Complexes and Units.
򐂰 Configure and manage Storage Facility Images.
򐂰 Create and delete RAID arrays, Ranks, and Extent Pools.
򐂰 Create and delete logical volumes.
򐂰 Manage host access to volumes.
򐂰 Check the current Copy Services configuration that is used by the Storage Unit.
򐂰 Create, modify, or delete Copy Services configuration settings.
4.2 Supported operating systems for the DS CLI

The DS Command-Line Interface can be installed on these operating systems:
򐂰 AIX 5.1, 5.2, 5.3
򐂰 HP-UX 11.0, 11i V1, V2
򐂰 HP-True64 5.1, 5.1A
򐂰 Linux (RedHat 3.0 Advanced Server (AS) and Enterprise Server (ES)
򐂰 SUSE Linux SLES 8, SLES 9, SUSE 8, SUSE 9
򐂰 Novell NetWare 6.5
򐂰 System i system i5/OS 5.3
򐂰 Sun™ Solaris 7, 8, 9
򐂰 Windows 2000, Windows Datacenter, Windows 2003, Windows XP
Note: The DS CLI cannot be installed on a Windows 64-bit operating system.
Important: For the most recent information about currently supported operating systems,
refer to the IBM System Storage DS6000 Information Center Web site:
http://publib.boulder.ibm.com/infocenter/ds6000ic/index.jsp
The DS CLI is supplied and installed via a CD that ships with the machine. The installation
does not require a reboot of the open systems host. The DS CLI requires Java™ 1.4.1 or

later. Java 1.4.2 for Windows, AIX, and Linux is supplied on the CD. Many hosts may already
have a suitable level of Java installed. The installation program checks for this requirement
during the installation process and does not install the DS CLI if you do not have the correct
version of Java.
The installation process can be performed via a shell, such as the bash or korn shell, or the
Windows command prompt, or via a GUI interface. If performed via a shell, it can be
performed silently using a profile file. The installation process also installs software that
allows the DS CLI to be completely de-installed should it no longer be required.
If you need any assistance to install the DS CLI you can refer to the publication IBM System
Storage DS8000 Command-Line Interface User´s Guide, SC26-7916.
4.3 User accounts

The admin account is set up automatically at the time of installation. It is accessed using the
user name admin and the default password admin. This password is temporary and you must
change the password before you can use any of the other functions. There are seven groups
the administrator can assign to a user. The groups and the associated functions allowed by
the assignment are as follows:
򐂰 admin: Allow access to all storage management console server service methods and all
Storage Image resources.
򐂰 op_volume: Allow access to service methods and resources that relate to logical volumes,
hosts, host ports, logical subsystems, and Volume Groups, excluding security methods.
򐂰 op_storage: Allow access to physical configuration service methods and resources,
including Storage Complex, Storage Image, Rank, Array, and Extent Pool objects.
򐂰 op_copy_services: Allow access to all Copy Services service methods and resources,
excluding security methods.
򐂰 service: Monitor authority, plus access to all management console server service methods
and resources, such as performing code loads and retrieving problem logs.
򐂰 monitor: Allow access to list and show commands. It provides access to all read-only,
nonsecurity management console server service methods and resources.
򐂰 no access: Does not allow access to any service method or Storage Image resources. By
default, this user group is assigned to any user account in the security repository that is
not associated with any other user group.
Note: A user can be assigned to more than one user group.
Important: When a new user is created, the password that was initially set will be
automatically expired and must be changed when the user first logs on.
4.4 DS CLI profile

You can create default settings for the command-line interface by defining one or more
profiles on the system. For example, you can specify the management console (MC) for the
session, specify the output format for list commands, specify the number of rows per page in
the command-line output, and specify that a banner is included with the command-line output.
Chapter 4. DS Command-Line Interface 25

If a user enters a value with a command that is different from a value in the profile, the
command overrides the profile.
You have several options for using profile files:

򐂰 You can modify the default profile. The default profile, dscli.profile, is installed in the profile
directory with the software, for example, c:\Program Files\IBM\DSCLI\profile\dscli.profile
for the Windows platform and /opt/ibm/dscli/profile/dscli.profile for UNIX® and Linux
platforms.
򐂰 You can make a personal default profile by making a copy of the system default profile as
<user_home>/dscli/profile/dscli.profile. The home directory <user_home> is designated
as follows:
– Windows system: C:\Documents and Settings\<user_name>
– Unix/Linux system: /home/<user_name>
򐂰 You can create a profile for the Storage Unit operations. Save the profile in the user profile
directory. For example:
– c:\Program Files\IBM\DSCLI\profile\operation_name1
– c:\Program Files\IBM\DSCLI\profile\operation_name2
Attention: The default profile file created when you install the DS CLI will potentially be
replaced every time you install a new version of the DS CLI. It is a better practice to open
the default profile and then save it as a new file. You can then create multiple profiles and
reference the relevant profile file using the -cfg parameter.
These profile files can be specified using the DS CLI command parameter -cfg
<profile_name>. If the -cfg file is not specified, the user’s default profile is used. If a user’s
profile does not exist, the system default profile is used.
Note: A password file generated using the managepwfile command is located at the
following directory: user_home_directory/dscli/profile/security/security.dat.
When you install the command-line interface software, the default profile is installed in the
profile directory with the software. The file name is dscli.profile, for example c:\Program
Files\IBM\DSCLI\profile\dscli.profile.
The available variables and detailed descriptions and information about how to handle them
can be found in the publication IBM System Storage DS8000 Command-Line Interface User´s
Guide, SC26-7916.
4.5 Command structure

This is a description of the components and structure of a command-line interface command.
A command-line interface command consists of one to four types of components, arranged in

the following order:
1. The command name: Specifies the task that the command-line interface is to perform.
2. Flags: Modify the command. They provide additional information that directs the
command-line interface to perform the command task in a specific way.
3. Flags parameter: Provides information that is required to implement the command
modification that is specified by a flag.

4. Command parameters: Provides basic information that is necessary to perform the
command task. When a command parameter is required, it is always the last component
of the command, and it is not preceded by a flag.
4.6 Copy Services commands

We can classify the commands available with the DS CLI as follows:
1. ls commands: Give you brief information about the Copy Services state. See Table 4-1.
2. show commands: Give you detailed information about the Copy Services state. See
Table 4-2.
3. mk commands: Used to create relationships. See Table 4-3.
4. rm commands: Used to remove relationships. See Table 4-4 on page 28.
5. options commands: Used to modify some options in relationships that were previously
created. See Table 4-5 on page 28.
Table 4-1 List commands

Command Description
lsflash Lists of FlashCopy relationships.
lsremoteflash Lists of Remote FlashCopy relationships (inband).
lspprc Lists of Remote Mirror and Copy volumes relationships.
lspprcpath Lists of existing Remote Mirror and Copy path definition.
lsavailpprcport Lists available ports that can be defined as Remote Mirror and Copy.
lssession Displays a list of Global Mirror sessions for an LSS.
Table 4-2 List detailed commands

Command Description
showgmir Displays detailed properties and performance metrics for Global Mirror.
showgmircg Displays Consistency Group status for a Global Mirror session.
showgmiroos Displays the number of unsynchronized (out of sync) tracks for a Global Mirror session.
Table 4-3 Creation commands

Command Description
mkflash Initiates a Point-in-Time Copy from a source to a target volume.
mkremoteflash Initiates a remote copy through a Remote Mirror and Copy relationship.
mkgmir Starts Global Mirror for a session.
mkpprc Establishes a Remote Mirror and Copy relationship.
mkpprcpath Establishes or replaces a Remote Mirror and Copy path over a Fibre Channel.
mkesconpprcpath Creates a Remote Mirror and Copy path over an ESCON connection.
mksession Opens a Global Mirror for a session.

Table 4-4 Deletion commands
Command Description
rmflash Removes a relationship between FlashCopy pairs.
rmremoteflash Removes a relationship between remote FlashCopy pairs.
rmgmir Removes Global Mirror for the specified session.
rmpprc Removes a Remote Mirror and Copy relationship.
rmpprcpath Removes a Remote Mirror and Copy path.
rmsession Closes an existing Global Mirror session.
Table 4-5 Options commands

Command Description
commitflash Commits data to a target volume to form a consistency between the source and target.
resyncflash Incremental FlashCopy process.
reverseflash Reverses the direction of a FlashCopy pair.
revertflash Overwrites new data with data saved at the last consistency formation.
setflashrevertible Modifies a remote FlashCopy pair that is part of a Global Mirror to revertible.
commitremoteflash Commits data to a target volume to form a consistency (inband).
resyncremoteflash Incremental FlashCopy process (inband).
revertremoteflash Overwrites new data with data saved at the last consistency formation (inband).
setremoteflashrevertible Modifies a remote FlashCopy pair that is part of a Global Mirror to revertible (inband).
unfreezeflash Resets a FlashCopy Consistency Group.
failoverpprc Changes a secondary device into a primary suspended and keeps the primary in current.
failbackpprc Usually used after a failoverpprc to reverse the direction of the synchronization.
freezepprc Creates a new Remote Mirror and Copy Consistency Group.
pausepprc Pauses an existing Remote Mirror and Copy volume pair relationship.
resumepprc Resumes a Remote Mirror and Copy relationship for a volume pair.
unfreezepprc Thaws an existing Remote Mirror and Copy Consistency Group.
pausegmir Pauses a Global Mirror processing for a session.
resumegmir Resumes a Global Mirror processing for a session.
chsession Allows you to modify a Global Mirror session.
Important: The Remote Mirror and Copy commands are asynchronous. This means that a
command is issued to the DS CLI server and if it is accepted successfully, you receive a
successful completion code; however, background activity may still be occurring. For
example, a Metro Mirror pair will take some time to establish, as the tracks need to be
copied across from the primary to secondary device. In this example you should check the
state of the volumes using the lspprc command until the pairs have reached the Duplex
state.

Note: The mkflash and mkpprc commands offer the -wait flag, which delays the command
response until copy complete status is achieved. You may choose to use this flag if you
wish to be sure of successful completion.
4.7 Using the DS CLI application

You have to log into the DS CLI application to use the command modes. There are three
command modes for the DS CLI:
򐂰 Single-shot mode
򐂰 Interactive mode
򐂰 Script mode
4.7.1 Single-shot mode

Use the DS CLI single-shot command mode if you want to issue an occasional command but
do not want to keep a history of the commands that you have issued.
You must supply the login information and the command that you want to process at the
same time. Use the following example to use the single-shot mode:
1. Enter:
dscli -hmc1 <hostname or ip address> -user <adm user> -passwd <pwd> <command>
2. Wait for the command to process and display the end results.
Example 4-1 shows the use of the single-shot command mode.
Example 4-1 Single-shot command mode

C:\Program Files\ibm\dscli>dscli -hmc1 10.10.10.1 -user admin -passwd adminpwd
lsuser
Date/Time: 24 de Maio de 2005 14h38min20s BRT IBM DSCLI Version: X.X.X.X
Name Group State
=====================
admin admin locked
admin admin active
exit status of dscli = 0
Note: When typing the command, you can use the host name or the IP address of the DS
HMC.
4.7.2 Script command mode

Use the DS CLI script command mode if you want to use a sequence of DS CLI commands.
Administrators can use this mode to create automated processes, for example, establishing
Remote Mirror and Copy relationships for volume pairs.
You can issue the DS CLI script from the command prompt at the same time that you provide
your login information:
1. Enter:
dscli -hmc1 <hostname ip address> -user <user name> -passwd <pwd> -script <full path of
script file>

2. Wait for the script to process and provide a report regarding the success or failure of the
process.
Example 4-2 shows the use of the script command mode.
Example 4-2 Script command mode

C:\Program Files\ibm\dscli>dscli -hmc1 10.10.10.1 -user admin -passwd adminpwd
-script c:\test.cli
Date/Time: 24 de Maio de 2005 14h40min22s BRT IBM DSCLI Version: X.X.X.X DS: IBM
IBM.1750-1367890
ID WWPN State Type topo portgrp
===============================================================
I0000 500507630E01FC00 Online Fibre Channel-LW SCSI-FCP 0
I0001 500507630E03FC00 Online Fibre Channel-LW - 0
============================================================================
- IBM.1750-1312345 IBM.1750-1312345 511 500507630EFFFC6F Online Enabled
Important : The DS CLI script can contain only DS CLI commands. Use of shell
commands results in process failure. You can add comments to the scripts prefixed by the
number sign (#).
When typing the command, you can use the host name or the IP address of the DS HMC.
4.7.3 Interactive mode

Use the DS CLI interactive command mode when you have multiple transactions to process
that cannot be incorporated into a script. The interactive command mode provides a history
function that makes repeating or checking prior command usage easy to do.
1. Log on to the DS CLI application at the directory where it is installed.
2. Provide the information that is requested by the information prompts. The information
prompts might not appear if you have provided this information in your profile file. The
command prompt switches to a dscli command prompt.
3. Begin using the DS CLI commands and parameters. You are not required to begin each
command with dscli because this prefix is provided by the dscli command prompt.
Example 4-3 shows the use of interactive command mode.
Example 4-3 Interactive command mode

C:\Program Files\ibm\dscli>dscli
Enter your username: admin
Enter your password:
Date/Time: 24 de Maio de 2005 14h42min17s BRT IBM DSCLI Version: X.X.X.X DS:
IBM.1750-1312345
.
dscli> lsarraysite
Date/Time: 24 de Maio de 2005 15h8min57s BRT IBM DSCLI Version: X.X.X.X DS: IBM.

1750-1312345
arsite DA Pair dkcap (Decimal GB) State Array
================================================
S1 0 146.0 Assigned A0
.
dscli> lssi
============================================================================
- IBM.1750-1312345 IBM.1750-1312345 511 500507630EFFFC6F Online Enabled
dscli> quit
Note: When typing the command, you can use the host name or the IP address of the DS
HMC.
4.8 Return codes

When the DS CLI is exited, the exit status code is provided. This is effectively a return code. If
DS CLI commands are issued as separate commands (rather than using script mode), then a
return code will be presented for every command. If a DS CLI command fails (for instance,
due to a syntax error or the use of an incorrect password), then a failure reason and a return
code will be presented. Standard techniques to collect and analyze return codes can be used.
The return codes used by the DS CLI are shown in Table 4-6.
Table 4-6 Return code table

Return code Category Description
0 Success The command was successful.
2 Syntax error There is a syntax error in the command.
3 Connection error There was a connection problem to the server.
4 Server error The DS CLI server had an error.
5 Authentication error Password or user ID details are incorrect.
6 Application error The DS CLI application had an error.
4.9 User assistance

The DS CLI is designed to include several forms of user assistance. The main form of user
assistance is via the help command. Examples of usage include:
򐂰 help lists all available DS CLI commands.
򐂰 help -s lists all DS CLI commands with brief descriptions of each.
򐂰 help -l lists all DS CLI commands with syntax information.

If the user is interested in more details about a specific DS CLI command, they can use -l
(long) or -s (short) against a specific command. In Example 4-4, the -s parameter is used to
get a short description of the mkflash command’s purpose.
Example 4-4 Use of the help -s command

dscli> help -s mkflash
mkflash The mkflash command initiates a point-in-time copy from source volumes to target
volumes.
In Example 4-5, the -l parameter is used to get a list of all parameters that can be used with
the mkflash command.
Example 4-5 Use of the help -l command

dscli> help -l mkflash
mkflash [ { -help|-h|-? } ] [-dev storage_image_ID] [-tgtpprc] [-tgtoffline] [-t
gtinhibit] [-freeze] [-record] [-persist] [-nocp] [-wait] [-seqnum Flash_Sequenc
e_Num] SourceVolumeID:TargetVolumeID ... | -
Man pages
A man page is available for every DS CLI command. Man pages are most commonly seen in
UNIX-based operating systems to give information about command capabilities. This
information can be displayed by issuing the relevant command followed by -h, -help, or -?, for
example:
dscli> mkflash -help
or
dscli> help mkflash
4.10 Usage examples

It is not the intent of this section to list every DS CLI command and its syntax. If you need to
see a list of all the available commands, or require assistance using DS CLI commands, you
are better served by reading the publication IBM System Storage DS8000 Command-Line
Interface User´s Guide, SC26-7916, or you can use the online help.
Example 4-6 shows some of the common commands used on a DS6000. (The same
commands are possible on the DS8000.)
Example 4-6 Examples of DS CLI commands

# The following command establishes flashcopy pairs
dscli> mkflash -dev IBM.1750-1300861 0100-0102:0300-0302
Date/Time: April 25, 2005 5:59:56 PM IST IBM DSCLI Version: 0.0.0.0 DS: IBM.1750-1300861
CMUC00137I mkflash: FlashCopy pair 0100:0300 successfully created.
# The following command establishes Remote Mirror and copy paths

dscli> mkpprcpath -dev IBM.1750-1300861 -remotedev IBM.1750-1300861 -remotewwnn
5005076303FFC03D -srclss 00 -tgtlss 02 I0030:I0031 I0100:I0101
Date/Time: 18:33:22 IST April 25, 2005 IBM DSCLI Version: 0.0.0.0 DS: IBM.1750-1300861
CMUC00149I mkpprcpath: Remote Mirror and Copy path 00:02 successfully established.
# The following command establishes Remote Mirror and copy pairs

dscli> mkpprc -dev IBM.1750-1300861 -remotedev IBM.1750-1300866 -type gcp 0006:0206
Date/Time: 18:43:31 IST April 25, 2005 IBM DSCLI Version: 0.0.0.0 DS: IBM.1750-1300861
CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 0006:0206 successfully
created.

Part 3
Part 3 FlashCopy
This part of the book describes the IBM System Storage FlashCopy when used in open
systems environments with the DS8000. We discuss the FlashCopy features and describe
the options for its setup. We also show which management interfaces can be used as well as
the important aspects to be considered when establishing FlashCopy relationships.
This part ends with examples showing how FlashCopy is used in the daily business to
support the various environments where immediate copies of production data are a
requirement.

5
Chapter 5. FlashCopy overview

FlashCopy creates a copy of a volume at a specific point-in-time, which we also refer to as a
Point-in-Time copy, instantaneous copy, or time-zero copy (t0 copy).
This chapter explains the basic characteristics of FlashCopy when used in an open systems
environment with the DS8000. The following topics are discussed:
򐂰 FlashCopy operational areas
򐂰 FlashCopy basic concepts
򐂰 FlashCopy in combination with other Copy Services
򐂰 FlashCopy in a storage LPAR environment

5.1 FlashCopy operational environments
It takes only a few seconds to establish the FlashCopy relationships for tens to hundreds or
more volume pairs. The copy is then immediately available for both read and write access. In
a 24x7 environment, the quickness of the FlashCopy operation allows us to use FlashCopy in
very large environments and to take multiple FlashCopies of the same volume for use with
different applications. Some of the different uses of FlashCopy are shown in Figure 5-1.
Production
Production Backup
Application
System System
Help desk
System or
Operation System
Reverse Operation
FlashCopy Data
Integration
Backup
System
System
Business Production System
Integration data Operation
Test Data Mining

System other System
systems
Development Analysis
Team
Figure 5-1 FlashCopy operational environments
FlashCopy is suitable for the following operational environments:

򐂰 Production backup system
A FlashCopy of the production data allows data recovery from an older level of data. This
might be necessary due to a user error or a logical application error. Let’s assume that a
user deleted accidentally a customer record. The production backup system could work
with a FlashCopy of the data. The necessary part of the customer data can be exported
and then be imported into the production environment. Thus, the production continues,
and while a specific problem is being fixed, the majority of the users can work with the
application without recognizing any problems.
The FlashCopy of the data could also be used by system operations to re-establish
production in case of any server errors.
򐂰 Data backup system
A FlashCopy of the production data allows the client to create backups with the shortest
possible application outage. The main reason for data backup is to provide protection in
case of source data loss due to disaster, hardware failure, software failure, or user errors.
򐂰 Data mining system
A FlashCopy of the data can be used for data analysis, thus avoiding performance impacts
for the production system due to long running data mining tasks.

򐂰 Test system
Test environments created by FlashCopy can be used by the development team to test
new application functions with real production data, thus speeding up the test setup
process.
򐂰 Integration system
New application releases (for example, SAP® releases) are likely to be tested prior to
putting them onto a production server. By using FlashCopy, a copy of the production data
can be established and used for integration tests.
With the capability to reverse a FlashCopy, a previously created FlashCopy can be used
within seconds to bring production back to the level of data it had at the time when the
FlashCopy was taken.
5.2 Terminology
When discussing about Metro Mirror, Global Copy and Global Mirror, you will see that the
following terms are frequently used and interchangeably:
򐂰 The terms local, production, application, primary, or source, denote the site where the
production applications run while in normal operation. These applications create, modify,
and read the data — the application data. The meaning is extended to the disk subsystem
that holds the data as well as to its components — volumes and LSS.
򐂰 The terms remote, recovery, backup, secondary, or target, denote the site to where the
data is replicated — the copy of the application data. The meaning is extended to the disk
subsystem that holds the data as well as to its components — volumes and LSS.
When discussing about FlashCopy we use the term source to refer to the original data that is
created by the application. And we use the term target to refer to the point-in-time backup
copy.
Also the terms LUN and volume are used interchangeably in our discussions.
5.3 Basic concepts

By doing a FlashCopy, a relationship is established between a source and a target. Both are
considered to form a FlashCopy pair.
As result of the FlashCopy either all physical blocks from the source volume are copied (full
copy) or — when using the nocopy option — only those parts that are changing in the source
data since the FlashCopy has been established. Currently, the target volume needs to be the
same size or bigger than the source volume whenever FlashCopy is used to flash a whole
volume.
Typically, large applications such as databases have their data spread across several
volumes and their volumes should all be FlashCopied at exactly the same point-in-time.
FlashCopy offers consistency groups, which allows multiple volumes to be FlashCopied at
exactly the same instance.
The following are basic characteristics of FlashCopy operation:

򐂰 Establish FlashCopy relationship
When the FlashCopy is started, the relationship between source and target is established
within seconds by creating a pointer table, including a bitmap for the target.
Chapter 5. FlashCopy overview 39

While the FlashCopy relationship is being created, the DS8000 holds off I/O activity to a
volume for a time period by putting the source volume in a SCSI queue full state. During
this state, any new I/Os issued receive a queue full error and are automatically reissued by
the host bus adapter. In this way, no user disruption or intervention is required. I/O activity
resumes when the FlashCopy is established.
If all bits for the bitmap of the target are set to their initial values, this means that no data
block has been copied so far. The data in the target is not modified during setup of the
bitmaps. At this first step, the bitmap and the data look as illustrated in Figure 5-2.
FlashCopy established at time t0 (time-zero)
time
t0
bitmap
1 1 1 1 1 1
source volume target volume
data data
t0 t0 t0 t0 t0 t0
Figure 5-2 FlashCopy at time t0
Once the relationship has been established, it is possible to perform read and write I/Os
on both the source and the target. Assuming that the target is used for reads only while
production is ongoing, things will look as illustrated in Figure 5-3.
Writing to the source volume and

reading from the source and the target volume
time
t0 tx ty tz
read 1 read 2
time-zero data not yet

available in target volume:
read write x write y write z
read it from source volume. bitmap
1 0 1 0 1 1
data data
t0 tx t0 tz t0 t0 t0 t0
before physical write to the source volume:

copy time-zero data from the source volume to the target volume
Figure 5-3 Reads from source and target volumes and writes to source volume

򐂰 Reading from the source
The data is read immediately. See Figure 5-3 on page 40.
򐂰 Writing to the source
Whenever data is written to the source volume while the FlashCopy relationship exists,
the storage subsystem makes sure that the time-zero-data is copied to the target volume
prior to overwriting it in the source volume.
To identify if the data of the physical block on the source volume needs to be copied to the
target volume, the bitmap is analyzed. If it identifies that the time-zero data is not available
on the target volume, then the data will be copied from source to target. If it states that the
time-zero data has already been copied to the target volume then no further action is
done. See Figure 5-3 on page 40.
It is possible to use the target volume immediately — for reading data and also for writing
data.
򐂰 Reading from the target
Whenever a read-request goes to the target while the FlashCopy relationship exists, the
bitmap is used to identify if the data has to be retrieved from the source or from the target.
If the bitmap states that the time-zero data hasn’t yet been copied to the target, then the
physical read is directed to the source. If the time-zero data has already been copied to
the target then the read will be performed immediately against the target. See Figure 5-3
on page 40.
򐂰 Writing to the target
Whenever data is written to the target volume while the FlashCopy relationship exists, the
storage subsystem makes sure that the bitmap is updated. This way the time-zero data
from the source volume never overwrites updates done directly to the target volume. See
Figure 5-4.
Writing to the target volume
time
t0 ta tb
write a write b
bitmap
0 0 1 0 1 1
data data
t0 tx t0 ty t0 t0 ta t0 tb
Figure 5-4 Writes to target volume
򐂰 Terminating the FlashCopy relationship

The FlashCopy relationship is automatically ended when all tracks have been copied from
the source volume to the target volume. The relationship can also be explicitly withdrawn
by issuing the corresponding commands. If the persistent FlashCopy option was specified
then the FlashCopy relationship must be withdrawn explicitly.

5.3.1 Full volume copy
When the copy option is invoked and the establish process completes, a background process
is started that copies all data from the source to the target. Once this process is finished and
if there were no updates on the target, the picture we get is similar to the one in Figure 5-5.
If not explicitly defined as persistent, the FlashCopy relationship ends as soon as all data is
copied.
Background copy
time
t0 tx ty
bitmap
0 0 0 0 0 0
data data
t0 tx t0 ty t0 t0 t0 t0 t0 t0 t0 t0
Background copy will copy all time-zero data from source volume to target volume
Figure 5-5 Target volume after full volume FlashCopy relationship finished
If there were writes to the target then the picture we get is similar to the one in Figure 5-6.
Background copy
time
t0 tx ty ta tb
bitmap
0 0 0 0 0 0
data data
t0 tx t0 ty t0 t0 ta t0 t0 tb t0 t0
Background copy will copy all time-zero data from source volume to target volume.
Figure 5-6 FlashCopy after updates to the target volume
5.3.2 Nocopy option

If FlashCopy is established using the nocopy option, then the result will be as shown in
Figure 5-3 on page 40 and Figure 5-4 on page 41. The relationship will last until it is explicitly
withdrawn or until all data in the source volume has been modified. Blocks for which no write

occurred on the source or on the target will stay as they were at the time when the FlashCopy
was established.
If the persistent FlashCopy option was specified, the FlashCopy relationship must be
withdrawn explicitly.
5.4 FlashCopy in combination with other Copy Services

Volume based FlashCopy can be used in various combinations with other Copy Services,
while the most suitable will depend on the characteristics of the environment and the
requirements.
Note: The sample scenarios discussed in the present section, only apply to full-volume
FlashCopy. Data set FlashCopy is not supported in the open systems environment.
5.4.1 FlashCopy and Metro Mirror
FlashCopy
source target
Metro
Mirror
primary
Metro primary secondary

Mirror
source target
FlashCopy
Figure 5-7 FlashCopy and Metro Mirror
As illustrated in Figure 5-7, at the primary Metro Mirror site, the following combinations are
supported:
򐂰 A FlashCopy source volume can become a Metro Mirror primary volume and vice versa.
The order of creation is optional.
򐂰 A FlashCopy target volume can become a Metro Mirror primary volume and vice versa. If
you wish to use a FlashCopy target volume as a Metro Mirror primary, be aware of the
following considerations:
– The recommended order is to first establish the Metro Mirror, and then create a
FlashCopy to that Metro Mirror primary using the -tgtpprc parameter.
The Metro Mirror secondary will not be in a fully consistent state until the Metro Mirror
enters the full duplex state.

– If you create the FlashCopy first and then do a Metro Mirror of the FlashCopy target,
you must monitor the progress of the FlashCopy background copy.
The Metro Mirror secondary will not be in a fully consistent state until the FlashCopy
background copy process is complete.
Use the copy option to ensure that the entire FlashCopy source volume data is copied
to the Metro Mirror secondary.
On the secondary site of the Metro Mirror, a FlashCopy source volume can be the Metro
Mirror secondary volume, and vice versa. There are no restrictions on which relationship
should be defined first.
5.4.2 FlashCopy and Global Copy
FlashCopy
source target
Global
primary
Copy
Global primary secondary

Copy
source target
FlashCopy
Figure 5-8 FlashCopy and Global Copy
As illustrated in Figure 5-8, at the primary Global Copy site the following combinations are
possible:
򐂰 A FlashCopy source volume can become a Global Copy primary volume and vice versa.
The order of creation is optional.
򐂰 A FlashCopy target volume can become a Global Copy primary volume and vice versa. If
you want to use a FlashCopy target volume as a Global Copy primary, be aware of the
following considerations:
– The recommended order is to first establish the Global Copy, and then create a
FlashCopy to that Global Copy primary using the -tgtpprc parameter.
The Global Copy secondary is will not be in a fully consistent state until the Global
Copy enters the full duplex state.
Execute the mkpprc -type mmir command to force the Global Copy to enter the full
duplex state.
– If you create the FlashCopy first and then do a Global Copy of the FlashCopy target,
you must monitor the progress of the FlashCopy background copy.

The Global Copy secondary will not be in a fully consistent state until the FlashCopy
background copy process is complete.
Use the copy option to ensure that the entire FlashCopy source volume data is copied
to the Global Copy secondary.
On the secondary site of the Global Copy a FlashCopy source volume can be based on the
secondary volume for the Global Copy.
5.4.3 FlashCopy and Global Mirror

FlashCopy in combination with Global Mirror supports only one type of relationship at the
primary site (see Figure 5-9).
FlashCopy
source target
secondary Global
Mirror
primary for
Global open
Mirror for open
Figure 5-9 FlashCopy and Global Mirror
A FlashCopy source volume can become a Global Mirror primary volume and vice versa. The
relationships can be established in any sequence. A FlashCopy target volume cannot
become a Global Mirror primary volume.
On the Global Mirror secondary site, the Global Mirror target volume cannot be used as
FlashCopy source or FlashCopy target unless the Global Mirror pair is first suspended.

5.5 FlashCopy in a DS8300 storage LPAR environment
The IBM DS8300 supports the LPAR technology for storage disk subsystems. This allows a
DS8300 model to be partitioned in two virtual storage systems. Each partition is called a
DS8000 storage LPAR or Storage Facility Image (SFI) — shortly, storage LPAR or storage
image, respectively. See Figure 5-10.
Processor Processor
complex 0 complex 1
Storage
LPAR01 Image 1 LPAR11
Storage
LPAR02 Image 2 LPAR12
Figure 5-10 DS8300 Storage Facility Image (SFI)
Note in the figure that two processor complex LPARs, one from each processor complex,
make one storage LPAR. That is, processor complex LPAR 01 and LPAR 11 make the
DS8000 storage LPAR 1 — storage image 1.
FlashCopy within the same SFI

A FlashCopy can always be established between volumes belonging to the same Storage
Facility Image (SFI). FlashCopy is not supported from a source volume in one DS8300
storage LPAR to a target volume on another storage LPAR.

6
Chapter 6. FlashCopy options

This chapter discusses the options of FlashCopy when working with the IBM System Storage
DS8000 series in an open systems environment. The following options are explained:
򐂰 Multiple Relationship FlashCopy.
򐂰 Consistency Group FlashCopy.
򐂰 FlashCopy on existing Metro Mirror or Global Copy source.
򐂰 Incremental FlashCopy.
򐂰 Remote FlashCopy.
򐂰 Persistent FlashCopy.
򐂰 Reverse Restore and Fast Reverse Restore.

6.1 Multiple Relationship FlashCopy
It is possible to establish up to 12 FlashCopy relationships using the same source. In other
words, a source volume can have up to 12 target volumes. However, a target volume can still
only have one source. Furthermore, cascading FlashCopy is not allowed (that is, a volume
cannot be both a source and a target volume).
Following is a summary, of the considerations that apply:

򐂰 A FlashCopy source volume can have up to 12 FlashCopy target volumes.
Note: Only one of those targets can be defined as incremental FlashCopy.
򐂰 For each source volume, only one FlashCopy relationship can be reversed (the one having
the -record attribute).
򐂰 A FlashCopy target volume can have only one FlashCopy source volume.
򐂰 A FlashCopy target volume cannot be a FlashCopy source volume at the same time.
Figure 6-1 illustrates what is possible and what is not with multiple relationship FlashCopy.
FlashCopy
source target
...
maximum is 12
...
a source can have up to

12 targets
FlashCopy FlashCopy
source target
source and
source target target
not allowed not allowed
a target can have only a volume or dataset can be only a

one source source or target at any given time
Figure 6-1 Multiple Relationship FlashCopy possibilities
Note: At any point-in-time, a volume or LUN can be only a source or a target.
6.2 Consistency Group FlashCopy

Applications might have their data spread over multiple volumes. In this case if FlashCopy
needs to be used for multiple volumes, they all have to be at a consistent level. Consistency
groups can be used to help create a consistent Point-in-Time copy across multiple volumes,
and even across multiple DS8000 storage systems, thus managing the consistency of
dependent writes.

With the DS CLI, you can establish a Consistency Group by using the freeze option and
identifying all FlashCopy pairs and target volumes belonging to a group.
With the Freeze FlashCopy Consistency Group option, the DS8000 holds off I/O activity to a
volume for a time period by putting the source volume in a queue full state. Therefore, a time
slot can be created during which dependent write updates do not occur, and FlashCopy uses
that time slot to obtain a consistent point-in-time copy of the related volumes. I/O activity
resumes when all FlashCopies are established.
Dependent writes
If the start of one write operation is dependent upon the completion of a previous write, the
writes are dependent. Application examples for dependent writes are databases with their
associated logging files. For instance, the database logging file will be updated after a new
entry has been successfully written to a tablespace. The chronological order of dependent
writes to the FlashCopy source volumes is the basis to provide consistent data at the
FlashCopy target volumes. For a more detailed understanding of dependent writes and how
the DS8000 enables the creation of consistency groups, thus ensuring data integrity on the
target volumes, you can refer to the discussion in 12.4.1, “Data consistency and dependent
writes” on page 117.
6.3 FlashCopy target as a Metro Mirror or Global Copy primary

With this option the FlashCopy target volume can also be a primary volume for a Metro Mirror
or Global Copy relationship. A user may wish to use this capability to create both a remote
copy and a local copy of a production volume.
Figure 6-2 illustrates this capability. In this figure, the FlashCopy target and the Metro Mirror
(or Global Copy) primary are the same volume. They are displayed as two separate volumes
for ease of understanding.
Local site Remote site
source target
FlashCopy
primary secondary
Global Metro Global Metro

Copy Mirror Copy Mirror
Figure 6-2 FlashCopy target is Metro Mirror (or Global Copy) primary
It is possible to either create the FlashCopy relationship first or the Metro Mirror (or Global
Copy) relationship first. However, in general it is better to create the FlashCopy relationship
first to avoid the initial sending of unnecessary data across to the Metro Mirror (or Global
Copy) secondary.
Chapter 6. FlashCopy options 49

6.4 Incremental FlashCopy — refresh target volume
Incremental FlashCopy provides the capability to refresh a FlashCopy relationship, thus
refreshing the target volume.
Important: A refresh of the target volume always overwrites any writes previously written
to the target volume.
Restriction: If a FlashCopy source has multiple targets, an incremental FlashCopy

relationship can be established with one and only one target.
To perform an incremental FlashCopy, you must first establish the FlashCopy relationship
with the Start Change Recording and Persistent FlashCopy options enabled.
nothing to be
done as data
was already
copied from
no updates in source source to target no updates in target
and is identical
on both sides
updates took place

no updates in target volume
in source volume
current source
updates took place updates took place
data will be
in source volume in target volume
copied to target
updates took place
no updates in source
in target volume
Figure 6-3 Updates to the target volume caused by a refresh target FlashCopy
With Incremental FlashCopy, the initial FlashCopy — copy or nocopy — relationship between
a source and target volume is subject to the following (see Figure 6-3):
򐂰 FlashCopy with nocopy option
If the original FlashCopy was established with the nocopy option, then the bitmap for the
target volume will be reset, and of course, the updates on the target volumes are
overwritten. However, using the incremental option will automatically convert this
relationship to a copy relationship, and the background copy will begin.
򐂰 FlashCopy with copy option
If the original FlashCopy was established with the copy option (full volume copy), then the
updates that took place on the source volume since the last FlashCopy will be copied to
the target volume. Also, the updates done on the target volume will be overwritten with the
contents of the source volume.

When initializing a FlashCopy with Start Change Recording activated, a second and third
bitmap will be used to identify writes done to the source or the target volume (see Figure 6-4).
All three bitmaps are necessary for incremental FlashCopy:
򐂰 Target bitmap: This bitmap keeps track of tracks not yet copied from source to target.
򐂰 Source Change Recording bitmap: This bitmap keeps track of changes to the source.
򐂰 Target Change Recording bitmap: This bitmap keeps track of changes to the target.
These bitmaps allow subsequent FlashCopies to transmit only those blocks of data for which
updates occurred. Every write operation to the source or target volume will be reflected in
these bitmaps by setting the corresponding bit to 0.
Reads and writes with Start Change Recording option set
time
t0 tx ty tz ta read 1 read 2 t b tc
write y write a time-zero data write b write c
read write x write z not yet available
in target :
read it from
bitmap source.
bitmap
0 0 1 0 0 1 0 0 1 0 0 0
source target
data data
t0 tx t0 tz t0 t0 ta t0 t tb t0 t
before physical write to the source:

copy time-zero data from the source to the target
Figure 6-4 FlashCopy with Start Change Recording set

When the refresh takes place, the bitmap used for change recording is used to analyze which
blocks need to be copied from the source volume to the target volume (see Figure 6-5).
Refresh FlashCopy target volume
time
t0 tx ty tz ta tb t c t 0'
bitmap bitmap
0 0 1 0 0 1 0 0 1 0 0 0
source target
data data
t0 tx t0 tz t0 t0 t0 tx t0 tz t0 t0
needs to be copied as a
write occured on the target
update to the source
needs to be copied
update to the source
needs to be copied
needs to be copied as a
write occured on the target
bitmap
1 1 1 1 1 1 1 1 1 1 1 1
source target
data data
t0' t0' t0' t0' t0' t0' t 0' t 0'
Figure 6-5 Refresh of the FlashCopy target volume
After the refresh (which takes place only on the bitmap level) the new FlashCopy based on
time-0’ is active. The copy of the time-0’ data to the target is done in the background.
Tip: You can do the incremental copy at any time. You do not have to wait for the previous
background copy to complete.
6.5 Remote FlashCopy

With the DS CLI you can use commands to manage a FlashCopy relationship at a remote
site. The commands can be issued from the local site, and then they are transmitted over the
Metro Mirror or Global Copy links. This eliminates the need for a network connection to the
remote site solely for the management of FlashCopy.
The FlashCopy source volume at the remote site must be the secondary volume of the Metro
Mirror or Global Copy pair.

primary secondary
Global Metro Global Metro

Copy Mirror Copy Mirror
source target
FlashCopy
Figure 6-6 Remote FlashCopy
Figure 6-6 illustrates this capability. In this figure, the Metro Mirror (or Global Copy) secondary
and the FlashCopy source are the same volume. They are displayed as two separate volumes
for ease of understanding.
6.6 Persistent FlashCopy

With this option the FlashCopy relationship continues until explicitly removed (until the user
terminates the relationship using one of the interface methods). If this option is not selected,
the FlashCopy relationship will exist until all data has been copied from the source volume to
the target.

6.7 Reverse restore
With this option, the FlashCopy relationship can be reversed by copying over modified tracks
from the target volume to the source volume (see Figure 6-7). The background copy process
must complete before you can reverse the order of the FlashCopy relationship to its original
source and target relationship. Change recording is a prerequisite for reverse restore.

will become will become
target volume source volume
nothing to be done
as data was already
copied from source
no updates in source to target and is no updates in target
identical on both
sides
updates took place

data of previous no updates in target volume
in source volume
target (now source)
updates took place updates took place
will be copied to
in source volume in target volume
previous source
(now target) updates took place
no updates in source
in target volume
Figure 6-7 Reverse restore
The source and target bitmaps (illustrated in Figure 6-4 on page 51) are exchanged and then
handled as described with the Incremental FlashCopy option.
6.8 Fast reverse restore

This option is used with Global Mirror. If you specify this option, you can reverse the
FlashCopy relationship without waiting for the completion of the background copy of the
previous FlashCopy.

6.9 Options and interfaces
Now that we have discussed the options available with FlashCopy, in this section we see how
the DS Command-Line Interface (DS-CLI) and the DS Storage Manager (DS-SM) interfaces
support them; see Figure 6-8. Note that there are some options that cannot be invoked from
the DS-SM — they are indicated with a cross in Figure 6-8.
Interface DS front ends

Function DS SM DS CLI
Multiple relationship FlashCopy
Consistency Group FlashCopy
Target on existing Metro Mirror or

Global Copy primary
Incremental FlashCopy
Remote FlashCopy
Persistent Flashcopy
Reverse restore,
fast reverse restore
Figure 6-8 FlashCopy options and interfaces

7
Chapter 7. FlashCopy ordering and

activation
This chapter explains how to order and activate FlashCopy for the IBM System Storage
DS8000.

7.1 Ordering FlashCopy
FlashCopy is an optional feature of the IBM System Storage DS8000 series. This feature is
also referred to as the Point-in-Time Copy (PTC) licensed function.
Note: For a detailed explanation of the features involved and the considerations you must
have when ordering FlashCopy we recommend that you refer to the announcement letters:
򐂰 IBM System Storage DS8000 — Functions Authorization (IBM 2244)
򐂰 IBM System Storage DS8000 Series (IBM 2107)
IBM announcement letters can be found at:

http://www.ibm.com/products
All DS8000 series licensed functions are enabled, authorized, managed, activated, and
enforced based upon the physical capacity contained in a 2107 system.
Enablement
Licensed functions are enabled for a DS8000 through the selection of the appropriate
licensed function indicator feature number on the 2107 base unit. Particularly for FlashCopy,
the 2107 feature that must be ordered is the Licensed Function Indicator - Point-in-time copy
(PTC) #0720.
Authorization
To use FlashCopy you must acquire the corresponding DS8000 Series Function
Authorization (IBM 2244-PTC) with the adequate feature number (#72xx) to establish the
extent of IBM authorization: license size in terms of physical capacity. The DS8000 Series
Function Authorizations are for billing purposes only and establish the extent of IBM
authorization for use of a particular licensed function on the IBM System Storage DS8000
series (IBM 2107).
7.2 Activating FlashCopy

IBM System Storage DS8000 licensed functions are managed using the Disk Storage
Feature Activation (DSFA) application. Also, the DS8000 licensed functions are activated by
the installation of feature activation codes into the 2107 system.
The feature activation codes are made available by IBM and are obtained using the Disk
Storage Feature Activation (DSFA) application at:
http://www.ibm.com/storage/dsfa
7.2.1 Management
The Disk Storage Feature Activation (DSFA) application can be used to:
򐂰 Select the license scope.
򐂰 Assign the license value.
򐂰 Deactivate a licensed function.
The previous activities are referred to as the management activities that you can perform with
the DSFA application and in relation to the DS8000 licensed functions.

License scope
Licensed functions are activated and enforced within a defined license scope. License scope
refers to the type of storage, and therefore the type of servers, that the function can be used
with:
򐂰 Fixed Block (FB): The function can be used only with data from Fibre Channel-attached
servers.
򐂰 Count Key Data (CKD): The function can be used only with data from FICON-attached
servers.
򐂰 Both FB and CKD (ALL): The function can be used with data from all attached servers.
In particular, the license scope of FlashCopy is multiple: it can either be FB, CKD, or ALL. For
this reason when ordering FlashCopy you must plan ahead what your point-in-time copy
requirements will be. Example 7-1 illustrates a sample case.
Example 7-1 FlashCopy planning example

A DS8000 has a total physical capacity of 15 TB and that capacity will be configured as:
10 TB open systems (FB)
5 TB System z (CKD)
Then, these could be an example of the required license functions:
2244-OEL: Operating environment ==> 15 TB (equal to total machine capacity)
2244-PAV: Parallel access volumes ==> 5 TB (equal to CKD-configured capacity)
2244-RMZ: Remote mirror for z/OS ==> 5TB (equal to CKD-configured capacity)
FlashCopy could either be:

10TB if FlashCopy will be used only with open systems (FB) data (equal to
FB-configured capacity)
5TB if FlashCopy will be used only with System z (CKD) data (equal to CKD-configured
capacity)
15TB if FlashCopy will be used with both open systems (FB) and zSeries (CKD) data
(equal to total machine capacity)
If a licensed function (as is the case with FlashCopy) has multiple license scope options, you
will be required to select a license scope during the initial retrieval of the feature activation
code.
Note: The license scope is not selected while ordering function authorization feature
numbers. The feature numbers only establish the extent of IBM authorization (in terms of
physical capacity) regardless of the storage type.
You can also change the license scope using the DSFA application after a licensed function
has been activated. A new feature activation code will be generated, and when you install it
into the machine, the function will be activated and enforced using the newly selected license
scope. Only an increase in the license scope (changing FB or CKD to ALL) is a nondisruptive
activity. A lateral change (changing FB to CKD or changing CKD to FB) or a reduction of the
license scope (changing ALL to FB or CKD) is a disruptive activity and requires a machine
IML.
On the Model 9B2, the license scope does not need to be the same for each IBM System
Storage DS8300 LPAR. For example, with the FlashCopy license function you can select a
license scope of FB for Storage System LPAR 1 and a license scope of ALL for Storage
System LPAR 2. You can also change the license scope on a Storage System LPAR basis
and, if the change is disruptive, you will only need to reboot the affected Storage System
LPAR.
Chapter 7. FlashCopy ordering and activation 59

License value
License value refers to the extent of IBM authorization and is referenced as terabytes (TB) of
physical capacity.
On the DS8000 Models 931 and 932, license value assignment is an optional activity you can
perform using the DSFA application. On these models, the feature activation codes will
always be generated using the total license value for each licensed function, unless you
assign a license value of zero (0.0 TB).
On the Model 9B2, license value assignment is a required activity that you perform using the
DSFA application. On this model, the license value does not need to be the same for each
Storage System LPAR. For example, if you have a 15 TB total authorization level for the PTC
licensed function, you can assign a license value of 9.1 TB to Storage System LPAR 1 and a
license value of 5.9 TB to Storage System LPAR 2.
You can also change the license value on a Storage System LPAR basis. If the change is a
reduction in the license value, you may need to reboot the affected Storage System LPAR.
Deactivation
Assigning a license value of zero (0.0 TB) provides the ability to deactivate (disable) a
licensed function. A new feature activation code will be generated, and when you install it into
the machine, the function will be deactivated during the next IML of the machine.
You can also reactivate a deactivated licensed function by changing the license value to a
non-zero value. A new feature activation code will be generated, and when you install it into
the machine, the function will be activated. Reactivation is a nondisruptive activity.
When you deactivate a licensed function using the DSFA application, the installed function
authorization feature numbers remain on the machine record. This means that you do not
need to reacquire your function authorization feature numbers if you want to reactivate the
licensed function at a future date.
7.2.2 Activation
Activation refers to the client retrieval and installation of the feature activation code into the
2107 system.
The feature activation code is made available by IBM and obtained using the DSFA
application at:
Note: To access DSFA, you will be required to enter information about your 2107 machine.
You can find this information about the Storage Unit General Properties page in the IBM
System Storage DS8000 Storage Manager application.
The following information is needed for the download of the keyfile and should be prepared
during planning:
򐂰 Model
The model of the DS8000 can be taken from the order.
򐂰 Serial number of the DS8000
The serial number of the DS8000 can be taken from the front of the base frame (lower
right corner). Especially if several machines have been delivered, this is the only way to
obtain the serial number of a machine located in a specific point in the computer center.

򐂰 Machine signature
The machine signature can only be obtained using the DS SM or the DS CLI after
installation of the DS8000. In a Web browser enter the URL of the DS HMC, for example:
http://10.10.10.10:8452/DS8000
On the following window, select on the left side Real-time manager → Manage
hardware → Storage units. In the resulting window, select the storage unit you are
interested in from the list boxes and click Select Action → Properties (and Go). This
displays the properties of the selected DS8000 including the machine signature.
򐂰 Order confirmation code
The order confirmation code is printed on the DS8000 series order confirmation code
document, which is sent to the client’s contact person together with the delivery of the
machine or prior to delivery.
Chapter 7. FlashCopy ordering and activation 61

8
Chapter 8. FlashCopy interfaces

The setup of FlashCopy in an open systems environment can be done using different
interfaces. This chapter explains and gives examples of the interfaces that can be used for
FlashCopy management when FlashCopy is used with the IBM System Storage DS8000 in
an open systems environment.

8.1 FlashCopy management interfaces - overview
There are various interfaces available for the configuration and management of FlashCopy
when used in an open systems environment with the DS8000:
򐂰 DS Command-Line Interface (DS CLI)
򐂰 DS Storage Manager (DS SM) Graphical User Interface (GUI)
򐂰 TotalStorage Productivity Center for Replication (TPC for Replication)
򐂰 DS Open Application Programming Interface (DS Open API)
This chapter gives an overview of the DS CLI and the DS SM for FlashCopy management.
DS CLI and DS SM are also covered in Part 2, “Interfaces” on page 15.
TotalStorage Productivity Center for Replication provides management of DS8000 series

business continuance solutions, including FlashCopy, Metro Mirror, and Global Mirror. TPC
for replication is covered in Chapter 35, “IBM TotalStorage Productivity Center for Replication”
on page 539. TPC for Replication includes similar functions to Global Mirror Utility (GMU).
GMU users should consider migrating to TPC for Replication.
Also, the DS Open Application Programming Interface (DS Open API), which is a set of
application programming interfaces that are available to be integrated in programs, can be
used for FlashCopy management and control. The DS Open API is not covered in the present
book. For information about the DS Open API, refer to the publication IBM System Storage
DS Open Application Programming Interface Reference, GC35-0516.
FlashCopy control with the interfaces

Independently of the interface that is used, when managing FlashCopy the following basic
sequence takes place:
1. FlashCopy is initiated by means of an interface.
The initialization process of a FlashCopy takes only some seconds. At the end of this
process FlashCopy is established based on the given parameters. This means that all the
necessary meta structures have been established. No data has yet been copied.
2. FlashCopy runs in the background.
This is the moment when FlashCopy copies (in the background) the necessary data to
create the point-in-time copy. The parameters given at initialization time define how
FlashCopy will work in the background. They will also define which subsequent activities
can be performed with this FlashCopy.
3. FlashCopy terminates.
FlashCopy either terminates automatically if all tracks have been copied, or needs to be
terminated explicitly (by means of an interface) if it has been defined as persistent.
8.2 DS CLI and DS SM - commands and options

This section summarizes the commands and select options you can use when managing
FlashCopy at the local and remote sites.
The DS CLI can be used to execute both local FlashCopy and remote FlashCopy commands.
The DS SM user interface and the DS Open API do not support remote FlashCopy.

8.2.1 Local FlashCopy management
The commands you can use as well as the displayed panel’s actions and options you can
select when working with the DS8000 provided interfaces DS CLI and DS SM for local
FlashCopy management are listed in Table 8-1.
Local FlashCopy refers to the situation where the FlashCopy is local as opposed to a remote
FlashCopy. For a discussion of the characteristics of a remote FlashCopy refer to 6.5
“Remote FlashCopy” on page 52.
Table 8-1 Local FlashCopy using DS CLI and DS SM

Options Command with Select option
DS CLI with DS SM
Create a FlashCopy
Create a local FlashCopy mkflash Create
Work with an existing FlashCopy
Display a list of FlashCopy relationships lsflash Main panel
Modify a FlashCopy pair that is part of a Global Mirror setflashrevertible restorable, Record
relationship to revertible Change Wizard
Commit data to the target volume commitflash Commit changes
Increment an existing FlashCopy pair resyncflash resynch target

(prerequisites:
-record and
-persist)
Change the source-target-relationship A → B to B → A reverseflash reverse

relationship
Reestablish contents of target B by contents of source A revertflash Consistency

as it was during last consistency formation Groups currently
not supported
Reset a FlashCopy Consistency Group unfreezeflash Consistency

Groups currently
not supported
Run new background copy for persistent FlashCopy rmflash -cp background copy
Terminate FlashCopy
Remove local FlashCopy rmflash Delete
Is automatically removed as soon as all

data is copied and FlashCopy pair was
not established using -persist parameter
8.2.2 Remote FlashCopy management

The commands that can be used when working with the DS8000 provided interface DS CLI
for remote FlashCopy management are listed in Table 8-2 on page 66.
Chapter 8. FlashCopy interfaces 65

Table 8-2 Remote FlashCopy using DS CLI commands
Options Command with DS CLI
Create a FlashCopy
Create a remote FlashCopy mkremoteflash
Work with an existing FlashCopy
Display a list of FlashCopy relationships lsremoteflash
Modify a FlashCopy pair that is part of a Global Mirror setremoteflashrevertible

relationship to revertible
Commit data to the target volume commitremoteflash
Increment an existing FlashCopy pair resyncremoteflash

(prerequisites: -record and -persist)
Change the source-target-relationship A → B to B → A reverseremoteflash
Reestablish contents of target B by contents of source A revertremoteflash

as it was during last consistency formation
Reset a FlashCopy Consistency Group Not available as remote command
Run new background copy for persistent FlashCopy rmremoteflash -cp
Terminate FlashCopy
Remove local FlashCopy rmremoteflash
Is automatically removed as soon as all

data is copied and FlashCopy pair was
not established using the -persist
parameter
Note: Remote FlashCopy is not supported with the DS SM or DS Open API interfaces.
8.3 Local FlashCopy using the DS CLI

The DS CLI can be downloaded from the IBM Web site and then installed on a workstation. It
communicates with the DS HMC. For a detailed information about the DS CLI, refer to the
publication IBM System Storage DS8000 Command-Line Interface User´s Guide,
SC26-7916.

8.3.1 Parameters used with local FlashCopy commands
This section discusses the parameters that can be passed to FlashCopy when using the DS
CLI and what the results are.
DS CLI Commands
mkflash lsflash setflash commit resync reverse revert unfreeze rmflash
Parameters revertible flash flash flash flash flash
Source
freeze x x
Target
tgtpprc x x x
tgtoffline x x x x
tgtinhibit x x x
tgtonly x
Flashcopy pair
dev x x x x x x x x x
record x x x x
persist x x x x
nocp x
seqnum x x x x x x x x
source:target x x x x
fast x
source x x
cp x x
source LSS x
l x
s x
activecp x
revertible x
Command
wait x
quiet x
Figure 8-1 Overview of parameters used in DS CLI FlashCopy commands
Figure 8-1 summarizes the parameters and the corresponding DS CLI commands. When
FlashCopy receives these parameters, the following actions result:
򐂰 freeze: Consistency Group FlashCopy.
With the DS CLI, it is possible to establish a Consistency Group by using the -freeze
parameter and identifying all FlashCopy pairs and target volumes belonging to it.
򐂰 tgtpprc: establish target on existing Metro Mirror source.
When this option is selected, the target volume can be or become a source volume for a
Metro Mirror or Global Copy relationship.
򐂰 tgtoffline: permit FlashCopy to occur if target volume is not online for host access.
When setting up the relationship, a verification is done to see whether the target volume is
online or offline. This parameter only applies to CKD volumes. Table 8-3 summarizes the
actions that result depending on the target volume status.
Table 8-3 Resulting actions when tgtoffline parameter is used

Status of target Permit FlashCopy to Comments
volume —CKD only occur if target
volume is online
Volume is online Not selected FlashCopy definition is not possible.
Volume is online Selected FlashCopy definition is queued until target

volume goes offline.

Status of target Permit FlashCopy to Comments
volume —CKD only occur if target
volume is online
Volume is offline Selected or not FlashCopy is started immediately.

selected
򐂰 tgtinhibit: inhibit writes to target volume.

If FlashCopy is active, writes to the target volume are not allowed (inhibited).
򐂰 record: change recording.
Activating the option change recording during setup of a FlashCopy enables subsequent
refreshes to the target volume. To do so a second bitmap is created for the source
volume, which keeps track of all writes to the source. This bitmap can later be used to
refresh the target by only copying the updates from the source to the target.
򐂰 persist: persistent FlashCopy.
The FlashCopy relationship will continue to exist until explicitly removed by an interface
method. If this option is not selected, the FlashCopy relationship will exist until all data has
been copied from the source volume to the target.
򐂰 nocp: full volume background copy.
With the parameter nocp it is possible to indicate whether the data of the source volume
will be copied to the target volume in the background. If -nocp is not used, a copy of all
data from source to target takes place in the background. With -nocp selected, only
updates to the source volume will cause writes to the target volume. This way the
time-zero data can be preserved.
򐂰 seqnum: sequence number for FlashCopy pairs.
A number that identifies the FlashCopy relationship. Once used with the initial mkflash
command, it could be used within subsequent commands to refer to multiple FlashCopy
relationships.
򐂰 source:target: identification of source volume and target volume.
򐂰 fast: reverse FlashCopy before background copy is finished.
Allows you to issue the reverseflash command before the background copy is finished.
򐂰 cp: restrict command to FlashCopy relationships with background copy.
򐂰 sourceLSS: reset Consistency Group for source logical subsystems.
򐂰 s: display of FlashCopy pairs with lsflash command.
The shortened output of the lsflash command returned. Only the FlashCopy pair IDs
display.
򐂰 l: display additional FlashCopy information.
The standard output of the lsflash command is enhanced. The values for the copy
indicator, out-of-synch tracks, date created, and date synchronized all additionally display.
򐂰 activecp: selection of FlashCopy pairs with active background copy.
򐂰 revertible: selection of FlashCopy pairs with revertible attribute.
8.3.2 Local FlashCopy commands - examples

This section explains the DS CLI commands that you can use to manage FlashCopy and also
shows examples of their use.

Initiate FlashCopy using mkflash
With mkflash, a local FlashCopy can be established. Four coding examples for mkflash are
shown in Example 8-1.
Example 8-1 mkflash command examples

Script
mkflash -dev IBM.2107-7506571 -seqnum 01 0001:0101
Date/Time: July 11, 2005 6:29:51 PM CEST IBM DSCLI Version: 5.0.3.134 DS: IBM.2107-7506571
mkflash -dev IBM.2107-7506571 -record -seqnum 02 0002:0102
mkflash -dev IBM.2107-7506571 -persist -seqnum 03 0003:0103
mkflash -dev IBM.2107-7506571 -nocp -seqnum 04 0004:0104
Listing of the properties of the FlashCopies

lsflash -dev IBM.2107-7506571 0001-0004
ID SrcLSS SequenceNum Timeout ActiveCopy Recording Persistent Revertible SourceWriteEnabled TargetWriteEnabled BackgroundCopy
====================================================================================================================================
0001:0101 00 1 300 Disabled Disabled Disabled Disabled Enabled Enabled Enabled
0002:0102 00 2 300 Disabled Enabled Enabled Disabled Enabled Enabled Enabled
0003:0103 00 3 300 Disabled Disabled Enabled Disabled Enabled Enabled Enabled
0004:0104 00 4 300 Disabled Disabled Disabled Disabled Enabled Enabled Disabled
The following explanations apply to the cases presented in Example 8-1:

򐂰 Example 1: 0001 → 0101
The FlashCopy between volume 0001 and volume 0101 is established using the default
parameters. By default, the following properties are enabled: SourceWriteEnabled,
TargetWriteEnabled, BackgroundCopy (default if not specified differently using the -nocp
parameter). All other properties are disabled. The background copy takes place
immediately and once everything has been copied, the FlashCopy relationship is
automatically removed.
򐂰 Example 2: 0002 → 0102
The FlashCopy between volume 0002 and volume 0102 is established with the following
FlashCopy properties enabled: Recording, Persistent, and BackgroundCopy.
Note: The parameter -persist is automatically added whenever -record is used.
The background copy takes place immediately and the relationship remains as a
persistent relationship. Using other DS CLI commands it could be reversed and
re-synchronized.
򐂰 Example 3: 0003 → 0103
The FlashCopy between volume 0003 and volume 0103 is established with the following
FlashCopy properties enabled: Persistent and BackgroundCopy. The background copy
takes place immediately. Once the background copy has finished the FlashCopy
relationship will remain, because of the persistent flag.
򐂰 Example 4: 0004 → 0104
The FlashCopy between volume 0004 and volume 0104 is established with the -nocp
parameter. This means, no full volume background copy will be done — only the data

changed in the source is copied to the target prior to changing it. Over time, this could
result in the situation where all data is copied to the target — then the FlashCopy
relationship would end. It would also end after a background copy is initiated using the DS
SM. This way the relationship is temporarily persistent even though the property Persistent
is not activated.
Display existing FlashCopy relationships using lsflash

The command lsflash can be used to display FlashCopy relationships and its properties.
Parameters can be used with this command to identify the subset of FlashCopy relationships
to be displayed.
Example 8-2 shows a script with several lsflash commands and the output of the script (this
script is logically based on the example for mkflash).
Example 8-2 lsflash command examples

#--- Example 1
lsflash -dev IBM.2107-7506571 0004
====================================================================================================================================
0004:0104 00 4 300 Disabled Disabled Enabled Disabled Disabled Disabled Enabled
#--- Example 2
lsflash -dev IBM.2107-7506571 0001-0005
====================================================================================================================================
0003:0103 00 3 300 Disabled Enabled Enabled Disabled Disabled Disabled Enabled
0005:0105 00 5 300 Disabled Disabled Disabled Disabled Disabled Disabled Disabled
#--- Example 3
lsflash -dev IBM.2107-7506571 -l 0001-0005
ID SrcLSS SequenceNum Timeout ActiveCopy Recording Persistent Revertible SourceWriteEnabled TargetWriteEnabled
BackgroundCopy OutOfSyncTracks DateCreated DateSynced
=======================================================================================================================
0003:0103 00 3 300 Disabled Enabled Enabled Disabled Disabled Disabled
Enabled 0 Mon Jul 11 19:30:06 CEST 2005 Mon Jul 11 19:30:06 CEST 2005
0004:0104 00 4 300 Disabled Disabled Enabled Disabled Disabled Disabled
Enabled 0 Mon Jul 11 19:30:10 CEST 2005 Mon Jul 11 19:30:10 CEST 2005
0005:0105 00 5 300 Disabled Disabled Disabled Disabled Disabled Disabled
Disabled 50085 Mon Jul 11 19:30:13 CEST 2005 Mon Jul 11 19:30:13 CEST 2005
#--- Example 4
lsflash -dev IBM.2107-7506571 -s 0001-0005
ID
=========
0003:0103
0004:0104
0005:0105
#--- Example 5
lsflash -dev IBM.2107-7506571 -activecp 0001-0004
#--- Example 6
lsflash -dev IBM.2107-7506571 -record 0001-0004
====================================================================================================================================

#--- Example 7
lsflash -dev IBM.2107-7506571 -persist 0001-0004
====================================================================================================================================
#--- Example 8
lsflash -dev IBM.2107-7506571 -revertible 0001-0004
#--- Example 9
lsflash -dev IBM.2107-7506571 -cp 0001-0004
====================================================================================================================================
The following explanations apply to the cases presented in Example 8-2 on page 70:
򐂰 Example 1: list FlashCopy information for a specific volume.
In this example, the lsflash command shows the FlashCopy relationship information for
volume 0004, showing the status (enabled/disabled) of the FlashCopy properties.
򐂰 Example 2: list existing FlashCopy relationships information within a range of volumes.
In this example, the lsflash command shows the FlashCopy relationships information for
the range of volumes 0001 to 0005, showing the properties status (enabled/disabled).
򐂰 Example 3: list existing FlashCopy relationships with full information.
Using the -l parameter with the lsflash command displays the default output plus
information about the following properties: OutOfSyncTracks, DateCreated, and
DateSynced.
򐂰 Example 4: list ids of existing FlashCopy pairs within a volume range.
Using the -s parameter displays only the FlashCopy source and target volume IDs for the
specified range of volumes.
򐂰 Example 5: list FlashCopy relationships with active background copy running.
Using the parameter -activecp will display only those FlashCopy relationships within the
selected range of volumes for which a background copy is actively running. The output
format is the default output. In our example there were no active background copies.
򐂰 Example 6: list existing FlashCopy relationships with -record enabled.
Using the -record parameter will display only those FlashCopy relationships within the
selected range of volumes that were established with the -record parameter.
򐂰 Example 7: list existing FlashCopy relationships with the Persistent attribute enabled.
When using the parameter -persist, only those FlashCopy relationships within the range of
selected volumes for which the Persistent option is enabled will be displayed.
򐂰 Example 8: list existing FlashCopy relationships which are revertible.
When using the parameter -revertible, only those FlashCopy relationships within the range
of selected volumes for which the option Revertible is enabled will be displayed. There
were no revertible relationships in our example.
򐂰 Example 9: list existing FlashCopy relationships for which BackgroundCopy is enabled.

When using the parameter -cp only those FlashCopy relationships within the range of
selected volumes for which the BackgroundCopy option is enabled will be displayed.
Set an existing FlashCopy to revertible using setflashrevertible

The command setflashrevertible can be used to modify the revertible attribute of a
FlashCopy relationship that is part of a Global Mirror relationship.
The FlashCopy properties Recording and Persistent must be enabled to set a FlashCopy
relationship to revertible using this command. This command needs to be executed prior to
running a commitflash or revertflash.
Example 8-3 illustrates two situations when using the setflashrevertible command.
Example 8-3 setflashrevertible command examples

#------------------------------------------------------------
#--- script to set FlashCopy property Revertible to value enabled and display values afterwards
#------------------------------------------------------------
#--- Example 1
setflashrevertible -dev IBM.2107-7506571 0002:0102
CMUC00167I setflashrevertible: FlashCopy volume pair 0002:0102 successfully made revertible.
lsflash -dev IBM.2107-7506571 0000-0004
====================================================================================================================================
0002:0102 00 0 300 Disabled Enabled Enabled Enabled Enabled Enabled Enabled
#--- Example 2
CMUN03027E setflashrevertible: 0003:0103: FlashCopy operation failure: action prohibited by current FlashCopy state

򐂰 Example 1: set the FlashCopy relationship to revertible
This command will set the existing FlashCopy for source volume 0002 and target volume
0102 to revertible. Once the property Revertible is enabled, any subsequent commands
will result in an error message similar to the one displayed in Example 2.
򐂰 Example 2: error when trying to set FlashCopy relationship to revertible
When trying to set a FlashCopy relationship to revertible for which the property Recording
is disabled, an error will result. The script will end after this command with return code 2
and any other commands following the one that causes the error will not be executed.
Commit data to target using commitflash

The command commitflash can be used to commit data to a target volume to set consistency
between source and target. It is intended to be used in asynchronous remote copy
environments such as Global Mirror. Therefore, its usage is discussed in greater detail in
Part 6, “Global Mirror” on page 241, while this section discusses the basic usage of the
command.
Before the FlashCopy relationship can be committed, it needs to be made revertible. Typically,
this is done automatically by an application such as Global Mirror. However, it can also be set
manually.

Example 8-4 Commit command examples
#--- Example 1
lsflash -dev IBM.2107-7506571 0000-0005
====================================================================================================================================
setflashrevertible -dev IBM.2107-7506571 -seqnum 01 0001:0101 0005:0105
lsflash -dev IBM.2107-7506571 0000-0005
====================================================================================================================================
0001:0101 00 1 300 Disabled Enabled Enabled Enabled Enabled Disabled Enabled
0005:0105 00 1 300 Disabled Enabled Enabled Enabled Enabled Disabled Enabled
commitflash -dev IBM.2107-7506571 0001-0005
CMUC00170I commitflash: FlashCopy volume pair 0001:0001 successfully committed.
lsflash -dev IBM.2107-7506571 -l 0000-0005
====================================================================================================================================
0001:0101 00 1 300 Disabled Enabled Enabled Disabled Disabled Enabled Disabled
0005:0105 00 1 300 Disabled Enabled Enabled Disabled Disabled Enabled Disabled
#--- Example 2
commitflash -dev IBM.2107-7506571 0003:0103
CMUN03027E commitflash: 0003:0003: FlashCopy operation failure: action prohibited by current FlashCopy state

򐂰 Example 1: Commit the FlashCopy relationship.
This example shows the properties of the two FlashCopy relationships 0001:0101 and
0005:0105 using the lsflash command prior and after issuing the setflashrevertible
command. After the commitflash command is executed the properties of the two
FlashCopy relationships are listed again.
򐂰 Example 2: Error when trying to commit a FlashCopy relationship.
When trying to commit a FlashCopy relationship that isn’t revertible (property Revertible is
disabled) an error will be the result. The script will end after this command with return code
2 and any other commands following the one that causes the error will not be executed.
Increment FlashCopy using resyncflash

With the resyncflash command an existing FlashCopy relationship can be incremented. To
run this command, the FlashCopy relationship must have the options Recording and
Persistent enabled.
Tip: You do not have to wait for the background copy to complete before you do the
FlashCopy re-synchronization. The resyncflash command can be used at any time.
To make sure an existing FlashCopy relationship can be incremented multiple times, it is

necessary to repeat the -record and -persist parameters with the resyncflash command.

Example 8-5 shows examples where the resyncflash command is used.
Example 8-5 resyncflash command examples

#--- Example 1
mkflash -dev IBM.2107-7506571 -record -persist -seqnum 01 0001:0101 0005:0105
mkflash -dev IBM.2107-7506571 -record -seqnum 03 0003:0103
lsflash -dev IBM.2107-7506571 0000-0005
====================================================================================================================================
resyncflash -dev IBM.2107-7506571 -record -persist -seqnum 11 0001:0101 0005:0105
CMUC00168I resyncflash: FlashCopy volume pair 0001:0101 successfully resynchronized.
resyncflash -dev IBM.2107-7506571 seqnum 13 0003:0103
lsflash -dev IBM.2107-7506571 0000-0005
====================================================================================================================================
0003:0103 00 13 300 Disabled Disabled Disabled Disabled Disabled Disabled Enabled
#--- Example 2
lsflash -dev IBM.2107-7506571 0004
====================================================================================================================================
resyncflash -dev IBM.2107-7506571 -record -persist -seqnum 14 0004:0104
CMUN03027E resyncflash: 0004:0104: FlashCopy operation failure: action prohibited by current FlashCopy state
The following explanations apply to the examples shown in Example 8-5:

򐂰 Example 1: Increment a FlashCopy relationship.
In this example, three FlashCopy relationships are created with the -record and -persist
parameters. The resyncflash commands are executed using a different sequence
number, which overwrites the one of the current FlashCopy relationship. The sequence
number only changes if the resyncflash finishes successfully.
The resyncflash for the 0001:0101 and 0005:0105 relationships take place using the
-record and -persist parameter. Because the two parameters are omitted for the
0003:0103 FlashCopy relationship the two properties Recording and Persistent change to
disabled for this FlashCopy relationship. As soon as the background copy for the
0003:0103 FlashCopy relationship finishes then the FlashCopy relationship will terminate.

򐂰 Example 2: Error when trying to increment FlashCopy relationship.
When trying to increment a FlashCopy relationship for which the properties Recording and
Persistent are disabled an error will be the result. The script will end after this command
with return code 2 and any other commands following the one that caused the error will
not be executed.
Reverse source-target relationship using reverseflash

The command reverseflash can be used to change the direction of a FlashCopy relationship.
The former source becomes target and the former target becomes source. The data is copied
from the target to the source.
Example 8-6 illustrates examples of the use of the reverseflash command.
Example 8-6 reverseflash command examples
#--- Example 1
lsflash -dev IBM.2107-7506571 0000-0005
====================================================================================================================================
reverseflash -dev IBM.2107-7506571 -record -persist 0001:0101 0005:0105
CMUC00169I reverseflash: FlashCopy volume pair 0001:0101 successfully reversed.
reverseflash -dev IBM.2107-7506571 -record 0002:0102
reverseflash -dev IBM.2107-7506571 0003:0103
lsflash -dev IBM.2107-7506571 0000-0005
====================================================================================================================================
0103:0003 01 3 300 Disabled Disabled Disabled Disabled Disabled Disabled Enabled
#--- Example 2
reverseflash -dev IBM.2107-7506571 -record 0002:0102
lsflash -dev IBM.2107-7506571 0002
====================================================================================================================================
#--- Example 3
reverseflash -dev IBM.2107-7506571 -seqnum 12 -record 0102:0002
lsflash -dev IBM.2107-7506571 0002
====================================================================================================================================

The following explanations apply to the examples shown in Example 8-6 on page 75:
򐂰 Example 1: Reverse a FlashCopy relationship.
In this example, three FlashCopy relationships are created with the -record and -persist
parameters. The reverseflash commands are executed using a different sequence
number, which overwrites the one of the current FlashCopy relationship.
The reverseflash for the 0001:0101 and 0005:0105 relationships take place using the
-record and -persist parameter. Because the two parameters are omitted for the
0003:0103 FlashCopy relationship the two properties Recording and Persistent change to
disabled for this FlashCopy relationship. This will terminate the 0003:0103 FlashCopy
relationship as soon it is successfully reversed.
򐂰 Example 2: reverse a FlashCopy relationship multiple times
It is possible to reverse a FlashCopy relationship multiple times, thus recopying the
contents of the original FlashCopy target volume multiple times back to the original source
volume. In this example the 0002:0102 was reversed once as part of example 1. Then
changes are done to data residing on volume 0002. A subsequent reverseflash for
0002:0102 eliminates the changes done to 0002 and brings back the data from volume
0102 to volume 0002 as it was during the initial FlashCopy.
򐂰 Example 3: reestablish original FlashCopy direction reversing again
It is possible to reverse a FlashCopy relationship back again. In example 3 this is shown
for the reversed FlashCopy relationship 0102:0002. Reversing it a second time and
referring to it as FlashCopy pair 0102:0002 is similar to establishing a new FlashCopy for
the volume pair 0002:0102. In this case a sequence number provided with the
reverseflash is used to identify a new FlashCopy relationship.
Reset target to contents of last consistency point using revertflash

The command revertflash can be used to reset the target volume to the contents of the last
consistency point. Like the commitflash command, this command is intended to be used in
asynchronous environments like Global Mirror environments. Before this command can be
issued, the relationship must be made revertible, either automatically as with Global Mirror, or
manually using the setrevertible command. See Example 8-7.
Example 8-7 revertflash command example

#--- Example 1
mkflash -dev IBM.2107-7506571 -record -persist -seqnum 01 0001:0101
Date/Time: July 12, 2005 12:12:20 AM CEST IBM DSCLI Version: 5.0.3.134 DS: IBM.2107-7506571
mkflash -dev IBM.2107-7506571 -nocp -seqnum 04 0001:0104 0001:0105
lsflash -dev IBM.2107-7506571 0000-0005
====================================================================================================================================
revertflash -dev IBM.2107-7506571 0001
CMUC00171I revertflash: FlashCopy volume pair 0001:0001 successfully reverted.

In Example 8-7 on page 76, three FlashCopy relationships are created for one source
volume: 0001:0101, 0001:0104, and 0105. The revertflash command is executed for source
0001 and as the FlashCopy relationship 0001:0101 has the Recording and Persistent
property enabled this command refers to this FlashCopy relationship 0001:0101. Any updates
done to volume 0101 will be overwritten.
Run new background copy for persistent FlashCopy using rmflash

Additional background copies for persistent FlashCopy relationships can be created using the
rmflash command in combination with its -cp parameter. See Example 8-8.
Example 8-8 rmflash command to create a new background copy

#--- Example 1
rmflash -dev IBM.2107-7506571 -quiet -cp 0001:0101
CMUC00143I rmflash: Background copy process for FlashCopy pair 0001:0101 successfully started. The persistent relationship will not be removed.
In the example presented in Example 8-8, create new background copy of a persistent
FlashCopy relationship. The existing FlashCopy relationship 0001:0101 is used to create a
new background copy.
Remove local FlashCopy using rmflash

The command rmflash can be used to remove a FlashCopy relationship. Unlike other
commands, it does not return an error if the request runs for a FlashCopy relationship that
does not exist.
In scripts it should always be used with the -quiet parameter to avoid the confirmation prompt.
See Example 8-9.
Example 8-9 rmflash command example

#--- Example 1
rmflash -dev IBM.2107-7506571 -quiet 0001:0101
CMUC00140I rmflash: FlashCopy pair 0001:0101 successfully removed.
In Example 8-9, the existing FlashCopy relationship 0001:0101 is removed.
8.3.3 FlashCopy Consistency Groups

Typically, large applications such as databases have their data spread across several
volumes. In order to create a consistent copy or backup, a FlashCopy of all the volumes
should be done at exactly the same point-in-time. FlashCopy Consistency Groups are
designed to achieve this exact purpose.
Create FlashCopy Consistency Groups using mkflash

Use the mkflash command with the -freeze parameter to create a FlashCopy Consistency
Group. This command causes the DS8000 to briefly prevent I/O to the volumes in the
Consistency Group. During this time, any I/O that comes from the host will be returned with a
SCSI queue full error, which the host bus adapter will automatically retry. However, if the I/O is
frozen for an extended period of time, there will be application errors on the host. For this
reason, the Consistency Group should be unfrozen as quickly as possible.

Example 8-10 illustrates the creation of a FlashCopy Consistency Group that contains two
FlashCopy pairs.
Example 8-10 Creating FlashCopy Consistency Groups

dscli> mkflash -dev IBM.2107-7506571 -freeze 1500-1501:1502-1503
Date/Time: October 24, 2005 3:37:49 AM PDT IBM DSCLI Version: 5.0.5.6 DS: IBM.2107-7506571
Reset FlashCopy Consistency Group using unfreezeflash

The command unfreezeflash can be used to remove a Consistency Group for multiple
volumes, for which FlashCopy relations were established using the -freeze parameter. This
command removes the queue full condition and allows I/Os to continue on the source
volumes. See Example 8-11.
The unfreezeflash command is issued to the entire logical subsystem (LSS).
Example 8-11 unfreezeflash command example

#--- Example 1
unfreezeflash -dev IBM.2107-7506571/00
CMUC00172I unfreezeflash: FlashCopy consistency group for logical subsystem 00: successfully reset.
8.4 Remote FlashCopy using the DS CLI

Remote FlashCopy commands are similar to local FlashCopy commands. The remote
commands can be issued whenever a DS8000 mirroring takes place from one DS8000 to
another DS8000. In this situation the Fibre Channel links between the two DS8000 (that are
used for mirroring purposes) are also used to transmit the FlashCopy commands to the
remote DS8000. For detailed information about the DS CLI, refer to the publication IBM
System Storage DS8000 Command-Line Interface User´s Guide, SC26-7916.
8.4.1 Remote FlashCopy commands

The syntax of the remote FlashCopy commands is similar to the syntax of the local
FlashCopy commands. The commands themselves have almost similar names except for the
remote character string that you can see in the remote FlashCopy commands names.
Table 8-4 shows the similarity between both sets of commands — their actions also similar.
Table 8-4 Command names for local and remote FlashCopy

Local command Remote command Comments
mkflash mkremoteflash Establish FlashCopy
lsflash lsremoteflash List FlashCopy
setflashrevertible setremoteflashrevertible Set FlashCopy to revertible
commitflash commitremoteflash Commit FlashCopy on target
resyncflash resyncremoteflash Increment FlashCopy
reverseflash reverseremoteflash Switch source-target

Local command Remote command Comments
revertflash revertremoteflash Reset to last consistency point
unfreezeflash - Reset Consistency Group
rmflash rmremoteflash Remove FlashCopy
8.4.2 Parameters used in remote FlashCopy commands
DS CLI Commands
mkremote lsremote setremote commit resync reverse revert rmflash
flash flash flash remote remote remote remote
Parameters revertible flash flash flash flash
Source
freeze x x
Target
tgtpprc x x x
tgtoffline x x x x
tgtinhibit x x x
tgtonly x
Flashcopy pair
dev x x x x x x x x
record x x x x
persist x x x x
nocp x
seqnum x x x x x x x x
source:target x x x x
fast x
source x x
cp x x
source LSS
l x
s x
activecp x
revertible x
Command
conduit x x x x x x x x
quiet x
Figure 8-2 DS CLI remote FlashCopy commands and parameters
Most of the parameters for remote FlashCopy commands are similar to those for local
FlashCopy commands. Two major exceptions between local FlashCopy and remote
FlashCopy commands are:
򐂰 Each remote command has the parameter conduit to identify the link to pass the
command to the secondary DS8000.
򐂰 The local FlashCopy command unfreezeflash has no remote equivalent.
Figure 8-2 summarizes the parameters and the corresponding DS CLI commands that can be
used when doing remote FlashCopy.
The description of the parameters is similar to the description presented in 8.3.1 “Parameters
used with local FlashCopy commands” on page 67. In regard to the parameter conduit, which
only applies to remote FlashCopy, the following explanation applies: Within a remote mirror
environment, the FlashCopy commands are sent across the mirror paths, thus avoiding the
necessity of having separate network connections to the remote site solely for the

management of the remote FlashCopy. The -conduit parameter identifies the path to be used
for transmitting the commands to the remote site.
8.5 FlashCopy management using the DS SM

To use the DS SM front end GUI, a supported Web Browser needs to be installed on the
workstation. The DS SM communicates with the DS HMC.
To start using the DS SM, enter in your Web browser the IP address of the DS HMC, as
shown in Example 8-12. The password is set up by the administrator and needs to be
changed during the first use of the DS SM.
Example 8-12 Starting DS SM

http://<ip-address:>:8451/DS8000
or
https://<ip-address>:8452/DS8000
8.5.1 Initiate FlashCopy using Create
Figure 8-3 FlashCopy Create with DS SM
After you login to the DS SM, follow this sequence (refer to Figure 8-3):
1. Select Real-time manager.
2. Select Copy services.
3. Select FlashCopy.
4. Use the Storage complex, Storage unit, and Storage image pull-down windows to identify
the DS8000 for which you would like to initiate a FlashCopy. Then use the Resource type
and Specify LSS pull-downs to display the volumes you want to work with on that DS8000.
5. From the Select action pull-down, select Create.

The previous selection will give you several windows, one after the other, to define the
attributes that will be used with this FlashCopy:
1. Define relationship type
Select either A single source with a single target, or if you want to allow a Multiple
Relationship FlashCopy, select A single source with multiple targets.
2. Select source volumes
Select one or multiple source volumes by checking the box at the left of the volume
identification. For each of the source volumes, you will afterwards be presented with a
window to select one or more targets.
3. Select target volumes
Select the target volumes (a source cannot have more than 12 target volumes).
4. Select common options
Figure 8-4 shows the options that can be chosen. The data provided in this window will be
used for all defined FlashCopy pairs.
Figure 8-4 Common option window for FlashCopy
5. Verification
In the verification window (not shown), all information regarding the FlashCopy definition is
displayed. If it is not possible to establish the FlashCopy relationship (if, for example, a
volume was requested to be offline, but is not offline), then an informational message is
displayed on this window.
Comparison of parameters for initial FC using DS SM and the DS CLI

A FlashCopy can be defined by using the DS SM with a browser (selecting the Create for
FlashCopy) or by using the DS CLI (using the mkflash command).
Using the DS SM, it is possible to define FlashCopies that will be generated immediately. It is
currently not possible to store any FlashCopy tasks using the DS SM.

Table 8-5 shows the corresponding parameters for the various FlashCopy options when using
the DS CLI command mkflash and the DS SM FlashCopy Create selection.
Table 8-5 Comparison of options and parameters used for FlashCopy in DS CLI and DS SM
Options Parameter Parameter with Comments
with DS CLI DS SM FlashCopy create
command
mkflash
Options for the source volume
Multiple relationship FlashCopy list of Multiple target volumes can

source:target be selected during create
volumes
Consistency Groups for freeze - Currently not supported with DS

FlashCopy SM front end
Options for the target volume
FlashCopy target can be Metro tgtpprc Establish target on existing

Mirror or Global Copy primary Metro Mirror source
Permit target volume to be online tgtoffline Permit FlashCopy to occur if This option exists only for CKD
in z/OS environment target is online for host access volumes (z/OS)
Inhibit writes to target volume tgtinhibit - Resync target
Options for FlashCopy pair
Identification of FlashCopy pair dev or Selected in windows

source:target
Change Recording record Enable change recording
Persistent FlashCopy persist Make relationship persistent
Full volume background nocp Initiate background copy

FlashCopy
Sequence number seqnum Sequence number for this

relationship
8.5.2 Display properties of existing FlashCopy

To display the properties of existing FlashCopy relationships, log in to the DS SM user
interface window and follow this sequence (refer to Figure 8-5 on page 83):
3. Select FlashCopy.
4. Use the Storage complex, Storage unit, and Storage image pull-down windows to identify
the DS8000 for which you would like to initiate a FlashCopy. Then use the Resource type
and Specify LSS pull-downs to display the volumes you want to work with on that DS8000.
This will give you a list of all active FlashCopy relationships. By checking the box left of the
FlashCopy of interest, the Select Actions for this FlashCopy relationship are changed based
on the attributes of the FlashCopy relationship and the possible actions to perform on it. Using
the Select Action → Properties, all attributes of the selected FlashCopy are displayed.

Note: In case you are selecting multiple FlashCopy relationships, the Select Action →
Properties will not be presented. Select only one FlashCopy relationship to view its
properties.
Figure 8-5 FlashCopy display properties
This selection gives you a window with two folders:

򐂰 General
In this folder, all properties of the selected FlashCopy are presented (see Figure 8-6).
Figure 8-6 General folder with FlashCopy properties information

򐂰 Out-of-synch tracks
The window displaying the out-of-synch tracks can be used to monitor how the FlashCopy
performs in the background. A refresh interval can be set to refresh the display after a
preselected period of time. See Figure 8-7.
Figure 8-7 Display out-of-synch tracks
Properties display - DS CLI versus DS SM

Table 8-6 compares the differences in the way that the FlashCopy information is displayed
when requested either using the DS CLI command lsflash or an existing FlashCopy
relationship was selected for display using the DS SM front end.
Table 8-6 FlashCopy properties as displayed by the DS CLI versus the DS SM

Properties with DS CLI command lsflash Properties with DS SM FlashCopy property
Property Contents Property Contents
Source LSS
SrcLSS # Source LSS in From selection list

selection window
Sequence Number to which the FlashCopy belongs. Can also be used for Consistency Groups.
SequenceNum ### Sequence number ###

from overview window
Shows if a Background Copy is currently running. If the DS CLI returns ActiveCopy=disabled this
could be similar to out-of-synch tracks>0 or copy complete on the DS SM.
ActiveCopy Enabled or disabled Status in overview Copy complete or

window out-of-synch tracks>0
or background copy
running
Shows if recording was selected for both source and target volume to allow for later usage of resync.
Recording Enabled or disabled Change recording Yes or no
Shows if the FlashCopy is a persistent one.
Persistent Enabled or disabled Relationship will Yes or no

remain
Identifies if the FlashCopy relationship can be changed by copying the contents of the target volume
to the source volume

Properties with DS CLI command lsflash Properties with DS SM FlashCopy property
Revertible Enabled or disabled Restorable Yes or no
Identifies if the source can be used to write on it
SourceWriteEnabled enabled or disabled Source is Yes or no

write-inhibited
Identifies it the target can be used by another server to write on it in parallel to the FlashCopy taking
place
TargetWriteEnabled Enabled or disabled Target is Yes or no

write-inhibited
Identifies if a background copy was already done
BackgroundCopy Enabled or disabled Background copy Yes or no

initiated
using -l parameter with DS CLI
OutOfSyncTracks A number Out-of-sync tracks A number
DateCreated Date Created Date
DateSynced Date Last refresh Date
8.5.3 Reverse existing FlashCopy

To reverse an existing FlashCopy relationship you first display the list of active FlashCopy
relationships as described in 8.5.2 “Display properties of existing FlashCopy” on page 82.
Then by checking the box at the left of the FlashCopy of your interest, the available Select
Actions for this FlashCopy relationship will be shown. If the existing FlashCopy relationship
can be reversed, it is possible to choose the Select Action and then select Reverse
relationship.

The next window displayed is shown in Figure 8-8. It displays those properties of the existing
FlashCopy that might be changed during the reverse process. Changing the values of the
parameters and then clicking OK will start the reverse process for the FlashCopy.
Figure 8-8 Parameters or copy options to use with reverse action

8.5.4 Initiate background copy for a persistent FlashCopy relationship
To initiate a background copy for a persistent FlashCopy relationship, start displaying the list
of active FlashCopy relationships as described in 8.5.2 “Display properties of existing
FlashCopy” on page 82. Then check the box at the left of the FlashCopy of your interest: the
available Select Actions for this FlashCopy relationship display. Then choose Select
Action → Initiate background copy (see Figure 8-9).
Figure 8-9 Initiate background copy
The next window that is presented is shown in Figure 8-10. It prompts for the FlashCopy pairs
for which the background copy should run.
Figure 8-10 Prompt window for background copy

8.5.5 Resynchronize target
To re-synchronize a target volume start by displaying the list of active FlashCopy
relationships, as shown in Figure 8-11. Then check the box at the left of the FlashCopy you
want to re-synchronize. Doing so, the available Select Actions for this FlashCopy relationship
will be shown. Then select Resync target.
Figure 8-11 Resync target select option to re-synchronize FlashCopy relationship
The prompt window asks for more details for the resync request (see Figure 8-12).
Figure 8-12 Prompt window to detail resync request for FlashCopy relationship

8.5.6 Delete existing FlashCopy relationship
To delete an existing FlashCopy relationship start by displaying the list of active FlashCopy
relationships, described in 8.5.2 “Display properties of existing FlashCopy” on page 82. Then
check the box at the left of the FlashCopy relationship you want to terminate. Doing so, the
available Select Actions for this FlashCopy relationship will be shown. Then select Delete
(see Figure 8-13).
Figure 8-13 Delete select option to delete FlashCopy relationship
The next window is a prompt asking you to confirm the delete request (see Figure 8-14).
Figure 8-14 Prompt window to confirm delete request for FlashCopy relationship

9
Chapter 9. FlashCopy performance

This chapter discusses best practices when configuring FlashCopy for specific environments
or scenarios. The following topics are covered:
򐂰 FlashCopy performance overview
򐂰 FlashCopy establish performance
򐂰 Background copy performance
򐂰 FlashCopy impact to applications
򐂰 FlashCopy options
򐂰 FlashCopy scenarios

9.1 FlashCopy performance overview
Many parameters can affect the performance of FlashCopy operations. It is important to
review the data processing requirements and hence select the proper FlashCopy options.
This chapter examines when to use copy versus no copy and where to place the FlashCopy
source and target volumes/LUNs. We also discuss when and how to use incremental
FlashCopy, which should definitely be evaluated for use in most applications.
Note: This chapter is equally valid for System z volumes and Open Systems LUNs. In the
following sections of the present chapter only the terms volume or volumes will be used,
but the text is equally valid if the terms LUN and LUNs were used, unless otherwise noted.
Terminology
Before proceeding with the discussion on FlashCopy best practices, let us review some of the
basic terminology we use in this chapter.
򐂰 Server: In a DS8000, a server is effectively the software that uses a logical partition
(LPAR) and that has access to a percentage of the memory and processor resources
available on a processor complex. The DS8000 models 931 and 932 have one pair of
servers — Server 0 and Server 1, one on each processor complex — both integrated in a
single Storage Facility Image (SFI). The DS8000 model 9B2 can have two pairs of servers
— two on each processor complex — each pair integrating one of the two possible SFIs.
You can issue the lsserver command to see the available servers.
򐂰 Device Adapter (DA): A physical component of the DS8000 that provides communications
between the servers and the storage devices. The lsda command lists the available
device adapters.
򐂰 Rank: An array site made into an array, which is then made into a rank. For the DS8000 a
rank is a collection of eight disk drive modules (DDMs). The lsrank command displays
detailed information about the ranks.
9.1.1 Distribution of the workload - source and target volumes location

In general, you can achieve the best performance by distributing the load across all of the
resources of the DS8000. In other words, you should carefully plan your usage so that the
load is:
򐂰 Spread evenly across disk subsystems
򐂰 Within each disk subsystem, spread evenly across servers
򐂰 Within each server, spread evenly across device adapters
򐂰 Within each device adapter, spread evenly across ranks
Additionally, it is always best to locate the FlashCopy target volume on the same DS8000
server as the FlashCopy source volume. It is also good practice to locate the FlashCopy
target volume on a different device adapter (DA) than the source volume, but there are some
cases where this is really an unimportant decision.

Another choice available is whether to place the FlashCopy target volumes on the same
ranks as the FlashCopy source volumes. In general it is best not to place these two volumes
in the same rank. Refer to Table 9-1 for a summary of the volume placement considerations.
Tip: If the FlashCopy target volume is on the same rank as the FlashCopy source volume
the user runs the risk of a rank failure causing the loss of both the source and the target
volumes.
Table 9-1 FlashCopy source and target volume location

Server Device Adapter Rank
FlashCopy establish Same server Unimportant Different ranks

performance
Background copy Same server Different device adapter Different ranks

performance
FlashCopy impact to Same server Unimportant Different ranks

applications
Tip: To find the relative location of your volumes, you can use the following procedure.
1. Use the lsfbvol command to learn which Extent Pool contains the relevant volumes.
2. Use the lsrank command to display both the device adapter and the rank for each
Extent Pool.
3. To determine which server contains your volumes, look at the Extent Pool name. Even
numbered Extent Pools are always from Server 0, while odd numbered Extent Pools
are always from Server1.
9.1.2 LSS/LCU versus rank considerations

In the DS8000 it is much more meaningful to discuss volume location in terms of ranks and
not in terms of logical subsystem (LSS) or logical control unit (LCU). On the ESS 800 and
earlier IBM disk subsystems, the physical location of the volumes was described in terms of
the logical subsystem LSS/LCU. If there existed more than a single rank in the LSS/LCU,
then each of the ranks would have a range of volumes from that specific LSS/LCU.
The LSSs/LCUs in a DS8000 disk subsystem are logical constructs that are no longer tied to
predetermined ranks. Within the DS8000 the LSS/LCU can be configured to span one or
more ranks but is not limited to specific ranks. There can be individual ranks that contain
volumes from more than a single LSS/LCU, which was not possible before the introduction of
the DS8000.
9.1.3 Rank geometry

Finally, you can achieve a small performance improvement by using identical rank geometries
for both the source and target volumes. In other words, if the source volumes are located on a
rank with a 7+p/RAID-5 configuration, then the target volumes should also be located on a
rank configured as 7+p/RAID-5.
Chapter 9. FlashCopy performance 93

9.1.4 Incremental FlashCopy
This chapter focuses on FlashCopy performance best practices, but there are many other
business requirements that must be weighed when using FlashCopy. The designer should
carefully consider all aspects for the implementation of each specific solution and should
definitely evaluate the use of incremental FlashCopy for all FlashCopy applications.
9.2 FlashCopy establish performance

The FlashCopy of a volume has two distinct periods:
򐂰 The initial logical FlashCopy (also called establish)
򐂰 The physical FlashCopy (also called background copy)
The FlashCopy establish phase, or logical FlashCopy, is the period of time when the
microcode is preparing things, such as the bitmaps, necessary to create the FlashCopy
relationship so the microcode can properly process subsequent reads and writes to the
related volumes. During this logical FlashCopy period, no writes are allowed to the volumes.
After the logical relationship has been established, normal I/O activity is allowed to both
source and target volumes according to the options selected.
When there are a large number of volumes, the method used to invoke the FlashCopy
commands can influence the time it takes to complete the logical FlashCopy for all FlashCopy
pairs. An in-band invocation will source the commands to the microcode faster than an
out-of-band method in most cases. An in-band establish is one that uses the CCW
commands directly, like in z/OS.
The fastest out-of-band method is the DS CLI. The method to control or invoke FlashCopy
should be selected based upon the total solution requirements and not strictly on logical
FlashCopy performance considerations unless this is a critical performance issue. For open
systems, the DS CLI is the fastest invocation method.
Additionally, there is a modest performance impact to logical FlashCopy performance when

using incremental FlashCopy. In the case of incremental FlashCopy, the DS8000 must create
additional metadata (bitmaps). However, the impact is negligible in most cases.
Finally, the placement of the FlashCopy source and target volumes has an effect on the
establish performance. You can refer to the previous section for a discussion on this topic as
well as to Table 9-1 on page 93 for a summary of the recommendations.
9.3 Background copy performance

The background copy phase, or physical FlashCopy, is the actual movement of the data from
the source volume to the target volume. If the FlashCopy relationship was established
requesting the -nocp (no copy) option, then only write updates to the source volume will force
a copy from the source to the target. This forced copy is also called a collision.
Note: The term collision describes a forced copy from the source to the target because a
write to the source has occurred. This occurs on the first write to a track. Note that since
the DS8000 writes to non-volatile cache, there is typically no direct response time delay on
host writes. The forced copy only occurs when the write is de-staged onto disk.

If the copy option was selected, then upon completion of the logical FlashCopy establish
phase, the source will be copied to the target in an expedient manner.
If a large number of volumes have been established, then do not expect to see all pairs
actively copying data as soon as their logical FlashCopy relationship is completed. The
DS8000 microcode has algorithms that will limit the number of active pairs copying data. This
algorithm will try to balance active copy pairs across the DS8000 device adapter resources.
Additionally, the algorithm will limit the number of active pairs such that there remains
bandwidth for host or server I/Os.
Tip: The DS8000 gives higher priority to application performance than background copy
performance. This means that the DS8000 will throttle the background copy if necessary,
so that applications are not unduly impacted.
The recommended placement of the FlashCopy source and target volumes, regarding the
physical FlashCopy phase, was discussed in the previous section. Refer to Table 9-1 on
page 93 for a summary of the conclusions. For the best background copy performance, the
implementation should always place the source and target volumes in different ranks. There
are additional criteria to consider if the FlashCopy is a full box copy that involves all ranks.
Note: The term full box copy has the implication that all rank resources are involved in the
copy process. Either all or nearly all ranks have both source and target volumes, or half the
ranks have source volumes and half the ranks have target volumes.
For full box copies, you should still place the source and target volumes in different ranks.
When all ranks are participating in the FlashCopy, it is still possible to accomplish this by
doing a FlashCopy of volumes on rank R0 onto rank R1, and volumes on rank R1 onto rank
R0 (for example). Additionally, if there is heavy application activity in the source rank,
performance would be less affected if the background copy target was in some other rank that
could be expected to have lighter application activity.
If background copy performance is of high importance in your environment, you should use
incremental FlashCopy as much as possible. Incremental FlashCopy will greatly reduce the
amount of data that needs to be copied, and therefore greatly reduces the background copy
time.
9.4 FlashCopy impact to applications

One of the most important considerations when implementing FlashCopy is to achieve an
implementation that has minimal impact to the users’ applications performance.
Note: As already mentioned, the recommendations discussed in this chapter only consider
the performance aspects of a FlashCopy implementation. But FlashCopy performance is
only one aspect of an intelligent system design. You must consider all business require-
ments when designing a total solution. These additional requirements, together with the
performance considerations, will guide you when choosing FlashCopy options like copy or
no copy and incremental, as well as when making choices on source and target volumes
location.
The relative placement of the source and target volumes has a significant impact on the
application’s performance, which has been discussed in 9.1.1, “Distribution of the workload -
source and target volumes location” on page 92.

In addition to the relative placement of volumes, the selection of copy or no copy is also an
important consideration in regards to impact to application performance. Typically, the choice
of copy or no copy depends primarily on how the FlashCopy will be used and for what interval
of time the FlashCopy relationship exists. From a purely performance point of view, the choice
of whether to use copy or no copy depends a great deal on the type of workload. The general
answer is to use no copy, but this is not always the best choice. For most workloads, including
online transaction processing (OLTP) workloads, no copy typically is the preferred option.
However, some workloads that contain a large number of random writes and are not cache
friendly might benefit from using the copy option, which is typically the preferred option.
Another important performance consideration is whether to use incremental FlashCopy.

Under the correct circumstances, incremental FlashCopy should have no impact to the critical
online write updates and should significantly shorten the period of time needed to complete
the periodic background copy.
Note: The incremental FlashCopy resyncflash command does not have a -nocp (no copy)
option. Using resyncflash will automatically use the copy option, regardless of whether the
original FlashCopy was copy or no copy.
9.5 FlashCopy options - considerations

Incremental FlashCopy is only supported at the volume level. Though incremental may be
used with FlashCopy relationships established with either the copy or no copy option, there
are some differences in how the copy and no copy relationships affect the application write
updates.
The copy approach causes an initial copy of all source volumes, but then will only copy
changed tracks when the Incremental Resync command is invoked. The no copy approach
does not perform an initial copy of the source volumes when the Incremental Resync
command is invoked, but does copy on all first updates to source tracks since the preceding
FlashCopy. Again, this copy only occurs when that track is de-staged, so in many cases there
is no impact to application performance.
The designer needs to consider the consequences of bursts of background copy, initially and
following each resync (copy option), which might impact the application — versus continuous
impact, when there are collisions that require a source to target copy before the write update
can complete (no copy option).
If running incremental FlashCopy for an extended period of time, the copy option could very
well be the proper choice.
9.6 FlashCopy scenarios

This section describes four scenarios. These scenario discussions assume that the primary
concern is to minimize the FlashCopy impact to application performance.
9.6.1 Scenario #1: Back up to disk

In environments where the Recovery Time Objective (that is, how quickly you can return to
production after a failure) is of utmost importance, a FlashCopy backup to disk can help to
achieve an extremely fast restore time. As soon as the logical FlashCopy is complete, it is
possible to perform a reverse FlashCopy and restore your production data in seconds,
instead of the several hours it would normally take to retrieve the data from tape.

When backing up to disk, it is important to take the necessary steps to protect your data.
Remember that, until the background copy is complete, you still only have one physical copy
of the data, and that copy is vulnerable. Therefore, it is important to always establish the
FlashCopy with the copy option. Otherwise, in the unlikely event you have a failure of your
production volumes, you will also lose your backup.
One method for protecting against failure is to use multiple FlashCopy targets. FlashCopy
supports up to 12 targets per source volume. With this feature, it is possible to keep up to 12
versions of your production data (for example, a FlashCopy backup every two hours for one
day). Another method is to use incremental FlashCopy. Incremental FlashCopy copies only
the data that has changed since the last FlashCopy, so the background copy completes much
faster.
9.6.2 Scenario #2: Back up to tape

If you want to create a copy of the data only to subsequently back up that data to tape, then
FlashCopy with the no copy option is the preferred approach. Still there are some
implementations where the copy option is employed.
The backup to tape is normally done shortly after the logical FlashCopy relationships have
been established for all of the volumes that are going to be backed up to tape.
If you choose the no copy option, that is probably because the data being backed up to tape
is mostly coming from the FlashCopy source volumes. If this is the case, then the location of
the target volumes is less critical and might be decided by considerations other than
performance.
If you choose the copy option, that is probably because the data being backed up is coming
from the target volumes — assuming that the backup to tape does not start until the
background copy completes. If the backup starts sooner, the data could be coming from a
mixture of source volumes and target volumes. As the backup continues, more and more of
the data will come from the target volumes as the background copy moves more and more of
the data to the target volumes.
To have the least impact on the application and to have a fast backup to tape, we recommend
that the source volumes be evenly spread across the available disk subsystems and the disk
subsystems resources. Once the backup to tape is complete, make sure to withdraw the
FlashCopy relationship.
Tip: Withdraw the pairs as soon as the backup to tape is finished. This eliminates any
additional copying from the source volume, either due to copy or collisions.
These recommendations would be equally valid for a copy or no copy environment.
9.6.3 Scenario #3: FlashCopy during peak application activity
Tip: The recommended solution is to fully explore alternatives that would allow no
overlapping of FlashCopy activity with other peak application activity. If such alternatives
are not viable for whatever operative reasons, then consider the topics discussed in the
present section.
As discussed previously, the choice of whether to use copy or no copy depends mostly on
business requirements, but with regards to performance, it also depends a great deal on the

type of workload. This topic has been discussed in 9.4, “FlashCopy impact to applications” on
page 95.
In general, no copy is the preferred method, but you should also think about the following
considerations when choosing either copy or no copy:
򐂰 Using no copy. The argument here is that the impact caused by the I/O resulting from the
copy option is more significant than that of the no copy option, where less I/O activity
resulting from collisions occur. Still, since the background copy only occurs when the
writes are de-staged from non-volatile cache, there is typically negligible impact. If the
workload is cache friendly, then potentially all of the operations will be served from cache,
and there will be no impact from collisions at all.
򐂰 Using copy. The goal of using copy is to quickly complete the background copy and hence
the overlapping situations between FlashCopy and application processing ends sooner. If
copy is used, then all I/Os experience some degradation as they compete for resources
with the background copy activity. However, this impact may be somewhat less than the
impact to the individual writes that a collision causes.
If FlashCopy no copy is active during a period of application high activity, there could be a
high rate of collisions (that is, de-stages being delayed so that the track image can be read
and then written to the FlashCopy target track to preserve the point-in-time copy). The
de-stage delay could cause degradation of the performance for all writes that occur during
the delay de-stage periods. Note that it is only the first write to a track that would cause a
collision, and only when that write gets de-staged. The reads do not suffer the collision
degradation.
If using the copy option, also consider these tips:
򐂰 Examine the application environment for the highest activity volumes and the most
performance sensitive volumes.
򐂰 Consider arranging the FlashCopy order such that the highest activity and most
performance sensitive volumes are copied early and the least active and least
performance sensitive volumes are copied last.
The intent here is minimize the potential for collisions.
Tip: One approach to achieve a specified FlashCopy order would be to partition the
volmes into priority groups. Issue the appropriate FlashCopy commands for all volumes,
but use copy on only the highest priority group and no copy on all other groups. After a
specified period of time or after some observable event, issue FlashCopy commands to
the next highest priority group from no copy to copy. Continue in this manner until all vol-
umes are fully copied.
If a background copy is the desired end result and FlashCopy is to be started just before or
during a high activity period, consider the possibility of starting with no copy and converting to
copy after the high activity period has completed.
One might also want to examine the use of incremental FlashCopy in a high performance
sensitive activity period. Incremental FlashCopy automatically uses the copy option, so if the
no copy option was previously selected, using incremental FlashCopy may impact
performance by causing a full background copy.
If the incremental FlashCopy approach is chosen, it might be best to create a FlashCopy copy
relationship during a quiet time. To minimize the amount of data to be copied when taking the
desired point in time copy, schedule an incremental refresh sufficiently in advance of the

point-in-time refresh to complete the copy of the changed data. Finally, take the required
point-in-time copy with the incremental refresh at the required point in time.
9.6.4 Scenario #4: Ranks reserved for FlashCopy

Another configuration worth considering is the one where 50% of the ranks (capacity) are all
FlashCopy source volumes (and where the application write I/Os take place) and the
remaining 50% of the ranks (capacity) are all FlashCopy target volumes.
Such an approach has pros and cons. The disadvantage is the loss of 50% of the ranks for
normal application processing. The advantage is that FlashCopy writes to target volumes will
not compete against applications writing to the target volumes. This allows the background
copy to complete faster, and thus reduces the interference with application I/Os.
This is a trade-off that must be decided upon:

򐂰 Use all ranks for your application:
– Maximize normal application performance.
– FlashCopy performance is reduced.
򐂰 Use only half of the ranks for your applications:
– Maximize FlashCopy performance.
– Normal performance is reduced.
If planning a FlashCopy implementation at the disaster recovery (DR) site, two distinct
environments must be considered:
򐂰 DR mirroring performance with and without FlashCopy active
򐂰 Application performance if DR failover occurs
The solution should provide acceptable performance for both environments.

10
Chapter 10. FlashCopy examples

This chapter presents examples of the use of FlashCopy in the following scenarios:
򐂰 Fast setup of test systems or integration systems
򐂰 Fast creation of volume copies for backup purposes

10.1 Create test system or integration system
Test systems or integration systems are needed to perform application tests or system
integration tests.
As it is assumed that with test systems or integration systems many write operations will
occur over the time period involved, we recommend doing a copy in the background
environment.
10.1.1 One time test system

Assume that there is an application using one volume, and a test system needs to be created
to allow application tests or integration tests based on the contents of the production data. A
FlashCopy is set up to copy the data once. See Example 10-1.
Example 10-1 Create a one time test system

#--- remove existing FlashCopy relationships for volume 6100
#--- establish FlashCopy relationships for source volume 6100

#--- list FlashCopy relationships for volume 6100

lsflash -dev IBM.2107-7506551 -l 6100
The application typically needs to be quiesced or briefly suspended before executing the
FlashCopy. Also, some applications cache their data, so this data may need to be flushed to
disk, using application methods, prior to running the FlashCopy (not covered in our example).
10.1.2 Multiple setup of a test system with the same content

Assume that an application test is needed multiple times with the same setup of data. The
production volume is 6100 and the test volume is 6300. Volume 6101 is chosen as an
intermediate volume that gets its data once, copying it from the production volume using
FlashCopy. Then it is used as base for refresh of the test volume 6300.
Running Part 1 and Part 2 (see Example 10-2) as one job would not work. The job would fail,
as volume 6101 could not be the target for one FlashCopy relationship and the source for
another FlashCopy relationship at the same time. You must wait until the background copy for
6100 and 6101 finishes successfully before starting Part 2.
Alternatively, you can also issue the rmflash command with the -cp and -wait parameters.
These parameters cause the command to wait until the background copy is complete before
continuing with the next step.
Example 10-2 Create a one time test system

#=== Part 1: establish FlashCopy relationship
#--- remove, establish, list FlashCopy relationships

lsflash -dev IBM.2107-7506551 -l 6100-6400
#=== Part 2: establish FlashCopy 2 relationship
#=== 03:00 pm 6100 → 6300
#--- after the full volume copy of 6100 → 6101 finished
#--- establish relationship from 6101:6300
lsflash -dev IBM.2107-7506551 -l 6100-6300
Whenever the test environment needs to be reset to the original data, just run Part 2 of the
scripts or use the DS SM front-end to perform a FlashCopy.
10.2 Create backup

Using FlashCopy for backup purposes can be implemented in several manners, as discussed
in this section.
10.2.1 Create a FlashCopy for backup purposes without volume copy

Volumes that are the result of a FlashCopy can be used by a backup server to backup the
data to tape. As the backup process merely reads the data, one option could be to perform a
FlashCopy without physically copying all data to the target. As soon as the backup of the data
has finished, the FlashCopy relationship could be removed explicitly.
The following are the steps involved in this procedure (refer Example 10-3 and
Example 10-4):
1. Part 1: Establish FlashCopy volume A → volume B with the no copy option.
2. Run backup.
3. Part 2: Remove the FlashCopy relationship once volume backup completes.
Example 10-3 Create a backup copy

#--- establish FlashCopy relationships for source volume 6100


lsflash -dev IBM.2107-7506551 -l 6100
After the backup has been taken, remove the FlashCopy relationship if you don’t intend to use
it for other purposes. Thus, unnecessary writes can be avoided. See Example 10-4.
Example 10-4 Create a backup copy

#=== Part 2: remove FlashCopy relationships
Doing backup based on the target volume allows you to use a target multiple times.
In complex application environments (for example, SAP), FlashCopy is often used as part of
the backup solutions. Good examples for such solutions are the IBM Tivoli® Storage Manager
for Hardware modules, which integrate into the IBM Tivoli Storage Manager backup
infrastructure.
Chapter 10. FlashCopy examples 103

10.2.2 Incremental FlashCopy for backup purposes
To have the safety of a real physical copy without always copying the full volume is something
that can be achieved using the incremental FlashCopy. An initial full volume FlashCopy is
followed by subsequent incremental FlashCopies, which only copies the updates that took
place on the source volume. See Example 10-5.
Example 10-5 Create an initial FlashCopy ready for subsequent incremental FlashCopies
#--- establish FlashCopy relationships

mkflash -dev IBM.2107-7506551 -record -persist -seqnum 01 6100:6300

lsflash -dev IBM.2107-7506551 -l 6100
After the initial full volume copy, the following script supports the incremental copy of the
FlashCopy relationship, see Example 10-6.
Example 10-6 Create an Incremental FlashCopy

#=== Part 2: resynch FlashCopy relationship
resyncflash -dev IBM.2107-7506551 -record -persist -seqnum 01 6100:6300
lsflash -dev IBM.2107-7506551 -l 6100
10.2.3 Using a target volume to restore its contents back to the source
Logs may need to be applied to the target, and then the target volume is reversed to the
source volume.
For each source volume, one FlashCopy can exist with the -record and -persist attributes
set. Use this volume to refresh the source volume. To reverse the relationship, the data must
have been copied completely to the target before reversing it back to the source. To avoid a
situation where the full volume needs to be copied with each FlashCopy, the Incremental
FlashCopy should be used. Since logs may need to be applied to the target volume prior to
reversing it, the target volume should be write-enabled.
This example consists of the following steps:

򐂰 Part 1: Establish initial FlashCopy (see Example 10-7).
򐂰 Part 2: Establish Incremental FlashCopy (see Example 10-8 on page 105).
򐂰 Part 3: Reverse the relationship (see Example 10-9 on page 105).
Applying application or DB logs needs to be carefully considered as well.
Example 10-7 Run initial FlashCopy to support refresh of source volume

#=== Part 1: Establish incremental FlashCopy
#--- remove, establish, list FlashCopy relationships
mkflash -dev IBM.2107-7506551 -persist -record -tgtinhibit -seqnum 01 6100:6101
lsflash -dev IBM.2107-7506551 -l 6100
After the initial FlashCopy, the incremental copies can be done; see Example 10-8.
Example 10-8 Create an Incremental FlashCopy

#=== Part 2: Resynch FlashCopy relationship
resyncflash -dev IBM.2107-7506551 -record -persist -tgtinhibit -seqnum 01 6100:6101
lsflash -dev IBM.2107-7506551 -l 6100
The reverse of the FlashCopy is done using the reverseflash command; see Example 10-9.
Example 10-9 Reverse the volumes

#=== Part 3: Reverse FlashCopy relationship
reverseflash -dev IBM.2107-7506551 -persist -record -tgtinhibit -seqnum 01 6100:6101
lsflash -dev IBM.2107-7506551 -l 6100
Chapter 10. FlashCopy examples 105

Part 4
Part 4 Metro Mirror

This part of the book describes IBM System Storage Metro Mirror when used in open
systems environments with the DS8000. We discuss the characteristics of Metro Mirror and
describe the options for its setup. We also show which management interfaces can be used,
as well as the important aspects to be considered when establishing a Metro Mirror
environment.
This part ends with examples of Metro Mirror management and setup.
Note: Throughout this part of the book, in our discussions of Metro Mirror in open systems
environments, you will find that the term volume is used indistinctly from the term LUN. In
fact, you will see that the term volume is almost always used.

11
Chapter 11. Metro Mirror overview

This chapter explains the basic characteristics of Metro Mirror when used in open systems
environments with the DS8000.

11.1 Metro Mirror overview
Metro Mirror (previously known as synchronous Peer-to-Peer Remote Copy, or PPRC)
provides real-time mirroring of logical volumes between two DS8000s that can be located up
to 300 km from each other. It is a synchronous copy solution where write operations are
completed on both copies (local and remote site) before they are considered to be complete.
It is typically used for applications that cannot suffer any data loss in the event of a failure.
As data is synchronously transferred, the distance between the local and the remote disk
subsystems will determine the effect on application response time. Figure 11-1 illustrates the
sequence of a write update with Metro Mirror.
4
Server
Write acknowledge
write
1 Write to secondary
2
LUN or 3 LUN or
volume volume
Write complete
Primary Secondary
(source)
acknowledgment (target)
Figure 11-1 Metro Mirror
When the application performs a write update operation to a source volume, this is what
happens:
1. Write to source volume (DS8000 cache and NVS).
2. Write to target volume (DS8000 cache and NVS).
3. Signal write complete from the remote target DS8000.
4. Post I/O complete to host server.
The Fibre Channel connection between the local and the remote disk subsystems can be
direct, through a switch, or through other supported distance solutions (for example, Dense
Wave Division Multiplexor or DWDM).
11.2 Metro Mirror volume state
Volumes participating in a Metro Mirror session can be found in either of the following states:
򐂰 Copy pending: Volumes are in copy pending state after the Metro Mirror relationship is
established, but the source and target volumes are still out-of-sync. In that case, data still
needs to be copied from the source to the target volume of the Metro Mirror pair. This may
be the case immediately after a relationship is initially established, or re-established after
being suspended. The Metro Mirror target volume is not accessible when the pair is in
copy pending state.
򐂰 Full duplex: The full duplex state is the state of a volume copy pair whose members are in
sync, that is, both source and target volumes contain exactly the same data. The target
volume is not accessible when the pair is in full duplex.
򐂰 Suspended: Volumes are in the suspended state when the source and target storage
subsystems cannot communicate anymore, or when the Metro Mirror pair is suspended
manually. In this state, writes to the source volume are not mirrored onto the target
volume. The target volume becomes out-of-sync. During this time, Metro Mirror keeps a
bitmap record of the changed tracks in the source volume. Later, when the volumes are
re-synchronized, only the tracks that were updated will be copied.
򐂰 Target copy pending: This indicates that the source volume is unknown or cannot be
queried and the target state is copy pending.
򐂰 Target full-duplex: This indicates that the source volume is unknown or cannot be queried
and the target state is full duplex.
򐂰 Target suspended: This indicates that the source volume is unknown or cannot be queried
and the target state is suspended.
򐂰 Not remote copy pair: This indicates that the relationship is not a Metro Mirror pair.
򐂰 Invalid-state: This indicates that the relationship state is invalid.
11.3 Data consistency

In order to restart applications at the remote site successfully, the remote site volumes must
have consistent data. In normal operation, Metro Mirror keeps data consistency at the remote
site. However, in case of a rolling disaster type of situation, a certain mechanism is necessary
to keep data consistency at the remote site.
For Metro Mirror, consistency requirements are managed through use of the Consistency
Group option. You can specify this option when you are defining Metro Mirror paths between
pairs of LSSs or when you change the default LSS settings. Volumes or LUNs that are paired
between two LSSs whose paths are defined with the Consistency Group option can be
considered part of a Consistency Group.
Consistency is provided by means of the extended long busy (for z/OS) or queue full (for open
systems) conditions. These are triggered when the DS8000 detects a condition where it
cannot update the Metro Mirror target volume. The volume pair that first detects the error will
go into the queue full condition, such that it will not do any writes, and an SNMP trap message
will be issued. These messages can be used as a trigger for automation purposes that will
provide data consistency.
Data consistency and dependent writes are discussed in detail in 12.4, “Consistency Group
function” on page 116.
Chapter 11. Metro Mirror overview 111

Note: During normal Metro Mirror processing, the data on disk at the remote site is an
exact mirror of that at the local site. During or after an error situation this depends on the
options specified for the pair and the path. Remember that any data still in buffers or pro-
cessor memory is not yet on disk and so will not be mirrored to the remote site. A disaster
will then appear to be a similar situation to a power failure in the local site.
11.4 Rolling disaster

In disaster situations, it is unlikely that the entire complex will fail at the same moment.
Failures tend to be intermittent and gradual, and a disaster can occur over many seconds,
even minutes. Because some data may have been processed and other data lost in this
transition, data integrity on the target volumes is exposed. This situation is called a rolling
disaster. The mirrored data at the recovery site must be managed so that cross-volume or
LSS data consistency is preserved during the intermittent or gradual failure.
Metro Mirror itself does not offer a means of controlling this scenario — it offers the
Consistency Group and Critical attributes, which along with appropriate automation solutions
can manage data consistency and integrity at the remote site. The Metro Mirror volume pairs
are always consistent, due to the synchronous nature of Metro Mirror. However,
cross-volume or LSS data consistency must have an external management method. IBM
offers the GDPS® and eRCMF service offerings to deliver solutions in this area. Visit the IBM
Web site and see the Services and Industry Solutions page for more information.
11.5 Automation and management

Metro Mirror is a hardware mirroring solution. A volume (or LUN) is paired with a volume (or
LUN) in the remote disk subsystem. As the size of the environment grows, so does the
complexity of managing it. You need a means for managing the pairs, ensuring that they are
in duplex status, adding volume pairs as required, monitoring for error conditions, and more
importantly, for managing data consistency across LSS and across disk subsystems.
When planning a Metro Mirror environment, the following topics should be considered:
򐂰 Design of the automation solution
򐂰 Maintenance
򐂰 Testing
򐂰 Support
All these must be considered. You do not want to be in a situation where you are relying on
your mirroring implementation for data to be consistent in a disaster situation, only to find that
it has not worked, or perhaps worse, you not being aware that your data is not consistent.
IBM offers services and solutions for the automation and management of the Metro Mirror
environment. These include GDPS and eRCMF. Visit the IBM Web site and see the Services
and Industry Solutions page for more information.
There are also solutions available for integrating Metro Mirror into a cluster, like HACMP/XD
for AIX, or IBM Total Storage Continuous Availability for Windows (GDS). See Part 9,
“Solutions” on page 537.
12
Chapter 12. Metro Mirror options and

configuration
This chapter discusses the options available when using Metro Mirror in an open systems
environment with the DS8000. It also discusses the configuration guidelines that should be
considered when planning the Metro Mirror environment.

12.1 Basic Metro Mirror operation
Before discussing the options and configuration guidelines for Metro Mirror, let us review
some basic operations you will be performing with Metro Mirror.
Establish a Metro Mirror pair

This operation establishes the remote copy relationship between a pair of volumes, the
source (or local) and the target (or remote) that normally reside on different disk subsystems.
Initially the volumes will be in simplex state, and immediately after the pair is established they
transition to the copy pending state. After the data on the pair has been synchronized (both
volumes have the same data), the state of the pair becomes full duplex.
When establishing a Metro Mirror pair, the following additional options can be selected:
򐂰 No copy
This option does not copy data from the source to the target. This option presumes that
the volumes are already synchronized. The data synchronization is your responsibility and
the DS8000 does not check its synchronization.
򐂰 Target read
This option allows host servers to read from the target. For a host server to read from the
target, the pair (source-target) must be in a full duplex state. This parameter applies to
open systems volumes (it does not apply to System z volumes.
In the open system’s file system environment, even if an application reads data from a file
system, a SCSI write command may be issued to the LUN where the file system resides.
This is because some information, such as the last access time stamp, may be updated by
the file system. In this case, even if you specify this option, the operation may fail.
򐂰 Suspend after data synchronization
This option suspends the volume pairs after the data has been synchronized.
򐂰 Wait option
This option delays the command response until the volume pairs are in one of the final
states: simplex, full duplex, suspended, target full duplex, target suspended (until the pair
is not in the copy pending state). This parameter cannot be used with -type gcp or -mode
nocp.
򐂰 Reset reserve on target
This option allows the establishment of a Metro Mirror relationship when the target volume
is reserved by another host. This parameter can only be used with open system volumes.
If this option is not specified and the target volume is reserved, the command fails.
Suspend Metro Mirror pair

This operation stops copying data to the target and the pair transitions to the suspended
state. Because the source DS8000 keeps track of all changed tracks on the source volume,
you can resume the copy operations at a later time.
Resume Metro Mirror pair

This operation resumes a Metro Mirror relationship for a volume pair that was suspended, and
restarts transferring data. Only modified tracks are sent to the target volume because the
DS8000 keeps track of all changed tracks on the source volume after the volume becomes
suspended. When resuming a Metro Mirror pair, you can use the same options as when
initially establishing a Metro Mirror pair except for the no copy option.
Terminate Metro Mirror pair
This operation ends the Metro MIrror relationship between the source and target volumes.
12.2 Clustering
Since the disk attached to the server is being mirrored using Metro Mirror, this offers
improved opportunities for high availability solutions.
There are also solutions available for integrating Metro Mirror into a cluster, like HACMP/XD
for AIX, and IBM Total Storage Continuous Availability for Windows (GDS). For more
information refer to Part 9, “Solutions” on page 537.
12.3 Failover and Failback

The Metro Mirror Failover and Failback modes are designed to help reduce the time required
to synchronize Metro Mirror volumes after switching between the production and the recovery
sites.
In a typical Metro Mirror environment, processing will temporarily switch over to the Metro
Mirror remote site upon an outage at the local site. When the local site is capable of resuming
production, processing will switch back from the remote site to the local site.
At the recovery site, the Metro Mirror Failover function combines into a single task the three
steps involved in the switchover (planned or unplanned) to the remote site: Terminate the
original Metro Mirror relationship, then establish and suspend a new relationship at the
remote site. The state of the original source volume at the normal production site is
preserved. The state of the original target volume at the recovery site becomes a source
suspended. This design takes into account that the original source LSS may no longer be
reachable.
To initiate the switchback to the production site, the Metro Mirror Failback function, at the
recovery site, checks the preserved state of the original source volume at the production site
to determine how much data to copy back. Then either all tracks or only out-of-sync tracks
are copied, with the original source volume becoming a target full-duplex. In more detail, this
is how Metro Mirror Failback operates:
򐂰 If a volume at the production site is in the simplex state, all of the data for that volume is
copied back from the recovery site to the production site.
򐂰 If a volume at the production site is in the full-duplex or suspended state and without
changed tracks, only the modified data on the volume at the recovery site is copied back to
the volume at the production site.
򐂰 If a volume at the production site is in a suspended state and has tracks that have been
updated, then both the tracks changed on the production site and the tracks marked at the
recovery site will be copied back.
򐂰 Finally, the volume at the production site becomes a write-inhibited target volume. This
action is performed on an individual volume basis.
Chapter 12. Metro Mirror options and configuration 115

The switchback is completed with one more sequence of a Metro Mirror Failover followed by a
Metro Mirror Failback operation, both given at the now recovered production site. Figure 12-1
summarizes the whole process.
Primary site (A) Secondary site (B)

Normal operation Updates are transferred
site (A) production site
Source = A Target = B
site (B) recovery site full duplex full duplex
When planned/unplanned outage at (A):

Establish with PPRC Failover
At (B): Metro Mirror Failover
A full duplex Source = B
restart application processing
suspended
With site (A) recovered and operable.

Establish with PPRC Failback
At (B):
1. Metro Mirror Failback Target = A Source = B
2. stop application processing copy pending copy pending
Establish with PPRC Failover

With pairs in full duplex.
At (A): Source = A B full duplex
suspended
1. Metro Mirror Failover
2. start application processing
Establish with PPRC Failback
3. Metro Mirror Failback
Source = A Target = B
full duplex full duplex
Figure 12-1 Metro Mirror Failover and Failback sequence
In this manner, Metro Mirror Failover and Failback are dual operations. It is possible to
implement all site switch operations with two pairs of failover and failback tasks (one pair for
each direction).
Note: For a planned switchover from site (A) to site (B), and in order to keep data consis-
tency at (B), the application at site (A) has to be quiesced before the Metro Mirror Failover
operation at (B). Alternatively, you can use the freezepprc and unfreezepprc commands.
The same consideration applies when switching back from site (B) to (A).
The Metro Mirror Failover and Failback modes can be invoked from the DS Storage Manager
GUI or the DS Command-Line Interface (CLI).
We give an example of a Metro Mirror Failover and Failback sequence and how to control it in
14.8, “Failover and Failback functions for sites switching” on page 147.
12.4 Consistency Group function

In order to restart applications at the remote site successfully, the remote site must have
consistent data. In normal operation, Metro Mirror keeps data consistency at the remote site.
However, as mentioned in 11.3, “Data consistency” on page 111, in case of a rolling disaster,
a certain procedure is necessary to keep data consistency even in a synchronous remote
copy environment.
In this section we explain data consistency and explain how the Metro Mirror Consistency
Group function keeps data consistency at the remote site in case of a rolling disaster.
12.4.1 Data consistency and dependent writes

Many applications, such as databases, process a repository of data that has been generated
over a period of time. Many of these applications require that the repository is in a consistent
state in order to begin or continue processing. In general, consistency implies that the order
of dependent writes is preserved in the data copy. In the Metro Mirror environment, keeping
data consistency means that the order of dependent writes is preserved in all the Metro Mirror
target volumes. For example, the following sequence might occur for a database operation
involving a log volume and a data volume:
1. Write to log volume: Data Record #2 is being updated.
2. Update Data Record #2 on data volume.
3. Write to log volume: Data Record #2 update complete.
If the copy of the data contains any of these combinations then the data is consistent:
򐂰 Operation 1, 2, and 3
򐂰 Operation 1 and 2
򐂰 Operation 1
If the copy of data contains any of those combinations then the data is inconsistent (the order
of dependent writes was not preserved):
򐂰 Operation 2
򐂰 Operation 3
When discussing the Consistency Group function, data consistency means this sequence is
always kept in the copied data. And the order of non-dependent writes does not necessarily
need to be preserved. For example, consider the following two sequences:
1. Deposit paycheck in checking account A
2. Withdraw cash from checking account A
3. Deposit paycheck in checking account B
4. Withdraw cash from checking account B
In order for the data to be consistent, the deposit of the paycheck must be applied before the
withdraw of cash for each of the checking accounts. However, it does not matter whether the
deposit to checking account A or checking account B occurred first, as long as the associated
withdrawals are in the correct order. So for example, the data copy would be consistent if the
following sequence occurred at the copy.
1. Deposit paycheck in checking account B
2. Deposit paycheck in checking account A
3. Withdraw cash from checking account B
4. WIthdraw cash from checking account A
In other words, the order of updates is not the same as it was for the source data, but the
order of dependent writes is still preserved.

12.4.2 Consistency Group function - how it works
In the operation of the Consistency Group function of Metro Mirror we distinguish two parts.
One is the invocation of the Consistency Group option and the other one is the freeze/run
operation. Together they make it possible for the disk subsystem to hold I/O activity and
subsequently to thaw the held I/O activities.
Consistency Group option

This option causes the disk subsystem to hold I/O activity to a volume for a time period by
putting the source volume into a queue full condition when the DS8000 detects a situation
where it cannot update the Metro Mirror target volume. This operation can be done across
multiple LUNs or volumes, multiple LSSs, and even across multiple disk subsystems. You
can specify this option when you are defining Metro Mirror paths between pairs of LSSs or
when you change the default Consistency Group setting on each LSS (the Consistency
Group option is disabled by default).
In the disk subsystem itself, each command is managed with each logical subsystem (LSS).
This means that there are slight time lags until each volume in the different LSS is subject to
the queue full condition. Some people are concerned that the time lag causes you to lose
data consistency, but that is not true. We explain how to keep data consistency in the
Consistency Group environments in the following section.
In the example in Figure 12-2, three write operations (first, second, and third) are dependent
writes. This means that these operations must be completed sequentially.
LSS11 LSS21
1st
Wait
Application
on
LSS12 LSS22
Servers
2nd
Wait
LSS13 LSS23
3rd
Wait
dependent write
operation
Source DS8000 Target DS8000
Figure 12-2 Consistency Group: Example1
In Figure 12-2, there are two Metro Mirror paths between LSS11 and LSS21. There are
another two Metro Mirror paths for each of the other LSS pairs (LSS12;LSS22 and
LSS13:LSS23). In a disaster, the paths may fail at different times. At the beginning of the
disaster, such as a rolling disaster, one set of paths (such as the paths between LSS11 and
LSS21) may be inoperable while other paths are working. At this time, the volumes in LSS11
are in a queue full condition, and the volumes in LSS12 and 13 are not. The first operation is
not completed because of the queue full condition, and the second and third operations are
not completed because the first operation has not been completed. In this case, the first,
second, and third updates are not included in the Metro Mirror target volumes in LSS21,
LSS22, and LSS23. Therefore, the Metro Mirror target volumes at the remote site keep
consistent data.
In the example illustrated in Figure 12-3, the volumes in LSS12 are in a queue full condition
and the other volumes in LSS11 and 13 are not. The first write operation is completed
because the volumes in LSS11 are not in an queue full condition. The second write operation
is not completed because of the queue full condition. The third write operation is also not
completed because the second operation is not completed. In this case, the first update is
included in the Metro Mirror target volumes, and the second and third updates are not
included. Therefore, this case is also consistent.
LSS11 LSS21
1st
Completed
Application
on
LSS12 LSS22
Servers
2nd
Wait
LSS13 LSS23
3rd
Wait
dependent write
operation
Source DS8000 Target DS8000
Figure 12-3 Consistency Group: Example 2
In all cases, if each write operation is dependent, the Consistency Group option can keep the
data consistent in the Metro Mirror target volumes until the Consistency Group time-out
occurs. After the time-out value has been exceeded, all held I/O will be released. The
Consistency Group time-out value can be specified at the LSS level.

If each write operation is not dependent, the I/O sequence is not kept in the Metro Mirror
target volumes that are in LSSs with the Consistency Group option specified. In the example
illustrated in Figure 12-4, the three write operations are independent. If the volumes in LSS12
are in a queue full condition and the other volumes in LSS11 and 13 are not, the first and third
operations are completed and the second operation is not completed.
LSS11 LSS21
1st
Completed
Application
on
LSS12 LSS22
Servers
2nd
Wait
LSS13 LSS23
3rd
Wait Completed
independent
write Source DS8000 Target DS8000
operations
Figure 12-4 Consistency Group: Example.3
In this case, the Metro Mirror target volumes reflect only the first and third write operations,
not the second operation.
Typical database management software can recover its databases by using its log files if
dependent write operations are kept. At the same time you should not need to care whether
independent write operations are kept in sequence.
Freeze and unfreeze operations

The Consistency Group option itself can keep consistent data at the remote site, in case of a
rolling disaster, if all volumes go into the queue full condition within the time interval specified
in the Consistency Group time-out value. However, this is not always true. Therefore, we need
a command that allows us to hold the I/O activity to volumes other than the volumes that the
DS8000 itself detects as having an error condition. We also need a command that allows us
to release the held I/O without having to wait for the Consistency Group time-out, in order to
minimize the impact on the applications. These commands are freezepprc and
unfreezepprc, which can be issued at an LSS level (not volume level). These commands
are available using the DS CLI, and we discuss them in this section.
When the DS8000 detects a condition where it cannot update the Metro Mirror target volume,
the Metro Mirror source volume with the Consistency Group option becomes suspended and
enters the queue full condition. At the same time the DS8000 issues an SNMP notification
(Trap 202: source PPRC Devices on LSS Suspended Due to Error).
An automation program, triggered by the SNMP notification, can issue the freezepprc
command to all LSS pairs that have volumes related to the application. This command
causes all Metro Mirror source volumes in the LSSs with the Consistency Group option to
become suspended and enter the queue full condition. In addition, this operation removes the
Metro Mirror paths. Because all Metro Mirror source volumes become suspended and all
related paths are removed, further updates to the source volumes will not be sent to their
targets. Now you have consistent data at the remote site. It is necessary to have an
automation procedure like the one just described in order to have consistent data at the
remote site in case of a rolling disaster.
In case of partial link failures (such as in the case of Figure 12-2 on page 118), and in order to
rapidly resume application I/O processing at the local site, you can use the unfreezepprc
command to resume held I/Os to the affected LSSs. In general, you will issue the
unfreezepprc command after successful completion of the freezepprc command to all
related LSSs, which means that you have consistent data at the remote site.
Note: By partial links failure we mean (in this section) the situation where for a particular
LSS all of its links fail, while at the same time there are other LSSs with their links working.
When any particular LSS has part of its links not working, Metro Mirror can keep sending
data to the target volumes using the other available links in the LSS.
The default two-minute timer of the queue full condition gives the automation enough time to
issue a freezepprc command to the necessary LSSs. I/O resumes after the default
two-minute interval if a unfreezepprc command is not received first.
It is not possible to issue Consistency Group (freeze/unfreeze) type commands from the DS
Storage Manager GUI, but you can change the time-out values by using the DS GUI.
Important: The queue full condition is presented only for the source volume affected by
the error (in the case of path failures, multiple volumes will often be affected). Still, the
freeze operation is performed at the LSS level, causing all Metro Mirror volumes in that
LSS to go into suspended state with queue full condition and terminating all associated
paths. Therefore, when planning your implementation, you should consider not intermixing
volumes from different applications in an LSS pair that is part of a Consistency Group.
Otherwise the not-in-error volumes belonging to other applications will be frozen, too.
12.5 Metro Mirror paths and links

Metro Mirror pairs are set up between volumes in LSSs, usually in different disk subsystems,
and these are normally in separate locations. A path (or group of paths) needs to be defined
between the source LSS and the target LSS. These logical paths are defined over physical
links between the disk subsystems.

The physical link includes the host adapter in the source DS8000, the cabling, switches, or
directors, any wideband or long distance transport devices (DWDM, channel extenders,
WAN), and the host adapters in the target disk subsystem. Physical links can carry multiple
Metro Mirror logical paths, as shown in Figure 12-5.
Note: For Metro Mirror, the DS8000 supports Fibre Channel links only. To facilitate ease of
testing, the DS8000 does support Metro Mirror source and target on the same DS8000.
LSS 0 LSS 0
LSS 1 Physical LSS 1
LSS 2 Fibre Channel link LSS 2
LSS 3 LSS 3
: :
LSS 08 256 logical paths LSS 08
: per FCP link :
LSS nn LSS nn
Figure 12-5 Logical paths
Paths are unidirectional, that is, they are defined to operate in either one direction or the
other. Still, Metro Mirror is bidirectional. That is, any particular pair of LSSs can have paths
defined among them that have opposite directions. Each LSS holds both source and target
volumes from the other particular LSS. Also, opposite direction paths are allowed to be
defined on the same Fibre Channel physical link.
For bandwidth and redundancy, more than one path can be created between the same LSSs.
Metro Mirror will balance the workload across the available paths between the source and
target LSSs.
Note: Remember that the LSS is not a physical construct in the DS8000 — it is a logical
construct. Volumes in an LSS can come from multiple disk arrays.
Physical links are bidirectional and can be shared by other Metro Mirror pairs as well as other
remote mirror and copy functions, such as Global Copy and Global Mirror.
12.5.1 Fibre Channel links

A DS8000 Fibre Channel port can simultaneously be:
򐂰 Sender for Metro Mirror source
򐂰 Receiver for Metro Mirror target
򐂰 Target for Fibre Channel Protocol (FCP) hosts I/O from open systems and Linux on
zSeries
Although one FCP link would have sufficient bandwidth for most Metro Mirror environments,
for redundancy reasons we recommend configuring two Fibre Channel links between each
source and remote disk subsystem.
Dedicating Fibre Channel ports for Metro Mirror use guarantees no interference from host I/O
activity. This is recommended with Metro Mirror, which is time critical and should not be
impacted by host I/O activity. Each Metro Mirror port provides connectivity for all LSSs within
the DS8000 and can carry multiple logical Metro Mirror paths.
Note: If you want the same set of Fibre Channel links to be shared between Metro Mirror
and Global Copy or Global Mirror, you should consider the impact of the aggregate data
transfer. In general, we do not recommend sharing the FCP links used for Metro Mirror,
including the physical network links, with other asynchronous remote copy functions.
Metro Mirror FCP links can be directly connected, or connected by up to two switches.
Note: If you use channel extension technology devices for Metro Mirror links, you should
verify with the product’s vendor what environment (directly connected or connected with a
SAN switch) is supported by the vendor and what SAN switch is supported.
12.5.2 Logical paths

A Metro Mirror logical path is a logical connecting path between the sending LSS and the
receiving LSS. An FCP link can accommodate multiple logical paths.
Figure 12-6 shows an example where we have a 1:1 mapping of source to target LSSs, and
where the three logical paths are accommodated in over one physical link:
򐂰 LSS1 in DS8000-1 to LSS1 in DS8000-2
Alternatively, if the volumes in each of the LSSs of DS8000-1 map to volumes in all three
target LSSs in DS8000-2, there will be nine logical paths over the physical link (not fully
illustrated in Figure 12-6). Note that we recommend a 1:1 LSS mapping.
DS8000 1 DS8000 2
3-9 logical paths
LSS 1 LSS 1
1 logical path 1 logical path

LSS 2 switch LSS 2
Port Port
1 logical path 1 Link 1 logical path
LSS 3 LSS 3
Metro Mirror
paths
1 logical path 1 logical path
Figure 12-6 Logical paths over a physical link for Metro Mirror

Metro Mirror paths have certain architectural limits, which include:
򐂰 A source LSS can maintain paths up to a maximum of four target LSSs. Each target LSS
can reside in a separate DS8000.
򐂰 You can define up to eight logical paths per LSS-LSS relationship. Each path requires a
separate physical link.
򐂰 An FCP port can host up to 2048 logical paths. These are the logical and directional paths
that are made from LSS to LSS.
򐂰 An FCP physical link (the physical connection from one port to another port) can host up
to 256 logical paths.
򐂰 An FCP port can accommodate up to 126 different physical links (DS8000 port to DS8000
port through the SAN).
12.6 Bandwidth
Prior to establishing your Metro Mirror solution, you should establish what your peak
bandwidth requirement will be. This will help to ensure that you have enough Metro Mirror
links in place to support that requirement.
To avoid any response time issues you should establish the peak write rate for your systems
and ensure that you have adequate bandwidth to cope with this load, and allow for growth.
Remember that only writes are mirrored across to the target volumes.
Tools to assist with this are operating system dependent, like iostat. Refer to the IBM
Redbook The IBM TotalStorage DS8000 Series: Implementation, SG24-6786. Another
method, but not so exact, is to monitor the traffic over the FC switches using FC switch tools
and other management tools, and remember that only writes will be mirrored by Metro Mirror.
You can also get some feeling about the proportion of read/writes by issuing datapath query
devstats on SDD-attached servers.
12.7 LSS design

Since the DS8000 has made the LSS a topological construct, which is not tied to a physical
array as in the ESS, the design of your LSS layout can be simplified. It is now possible to
assign LSSs to applications, for example, without concern regarding under-allocation or
over-allocation of physical disk subsystem resources.
This can also simplify the Metro Mirror environment, as it is possible to reduce the number of
commands that are required for data consistency as well as make its effects more granular.
For example, a freeze operation is performed at the LSS level, causing all Metro Mirror
volumes in that LSS to go into suspended state with a queue full condition and terminate all
associated paths. If you assign LSSs to each of your applications, you can control the impact
of the queue full condition caused by the freeze operation at an application level. On the
contrary, if you put volumes used by several different applications into the same LSS, all those
applications sharing the LSS and their Metro Mirror volumes will be affected by the queue full
condition. You can issue one freezepprc command to multiple LSSs. Therefore, you do not
need to consolidate the number of LSSs that your applications use. See 12.4.2, “Consistency
Group function - how it works” on page 118 for more information.
12.8 Distance
The distance between your local and remote DS8000 subsystems has an effect on the
response time overhead of the Metro Mirror implementation. The maximum supported
distance for Metro Mirror is 300 km.
12.9 Symmetrical configuration

When planning your Metro Mirror configuration, consider the possibility of a symmetrical
configuration, in terms of both physical and logical elements. This will have the following
benefits:
򐂰 Simplified management. It is easier to see where volumes will be mirrored, and processes
can be easily automated.
򐂰 Reduced administrator overhead. Due to automation, and the simpler nature of the
solution, overhead can be reduced.
򐂰 Simplify the addition of new capacity into the environment. New arrays can be added in a
modular fashion.
򐂰 Ease problem diagnosis. The simple structure of the solution will aid in identifying where
any problems may exist.
Figure 12-7 on page 126 shows this idea in a graphical form. DS8000 #1 has Metro Mirror
paths defined to DS8000 #2, which is in a remote location. On DS8000 #1, volumes defined
in LSS 00 are mirrored to volumes in LSS 00 on DS8000 #2 (volume P1 is paired with volume
S1, P2 with S2, P3 with S3, and so on). Volumes in LSS 01 on DS8000 #1 are mirrored to
volumes in LSS 01 on DS8000 #2, and so on. Requirements for additional capacity can be
added in a symmetrical way also by addition of volumes into existing LSSs, and by the
addition of new LSSs when needed. (For example, addition of two volumes in LSS 03 and 05,
and one volume to LSS 04 will bring them to the same number of volumes as the other LSSs.
Additional volumes could then be distributed evenly across all LSSs, or additional LSSs
added.)

As well as making the maintenance of the Metro Mirror configuration easier, the symmetrical
implementation has the added benefit of helping to balance the workload across the DS8000.
Figure 12-7 shows a logical configuration. This idea applies equally to the physical aspects of
the DS8000. You should attempt to balance workload and apply symmetrical concepts to
other aspects of your DS8000 (for example, the Extent Pools).
Figure 12-7 Symmetrical Metro Mirror configuration
12.10 Volumes
You need to consider which volumes should be mirrored to the target site. One option is to
mirror all volumes. This is advantageous for the following reasons:
򐂰 You will not need to consider whether any required data has been missed or not.
򐂰 Users will not need to remember which logical pool of volumes is mirrored and which is
not.
򐂰 Adding volumes to the environment is simplified. You do not have to have two processes
for addition of disk (one for mirrored volumes, and another for non-mirrored volumes).
򐂰 You will be able to move data around your disk environment easily without worrying about
whether the target volume is a mirrored volume.
You may choose not to mirror all volumes. In this case, you will need careful control over what
data is placed on the mirrored volumes (to avoid any capacity issues) and what data you
place on the non-mirrored volumes (to avoid missing any required data). One method of
doing this is to place all mirrored volumes in a particular set of LSSs, in which all volumes are
Metro Mirror-enabled, and direct all data requiring mirroring to these volumes.
Though mirroring all volumes might be the simpler solution to manage, it could also require
significantly more network bandwidth. Since network bandwidth is a cost to the solution,
minimizing the bandwidth may well be worth the added management complexity.
12.11 Hardware requirements
Metro Mirror is an optional licensed function available on the DS8000. Licensed functions
require the selection of a DS8000 series feature number (IBM 2107) and the acquisition of
DS8000 series Function Authorization (IBM 2244) feature numbers:
򐂰 The 2107 licensed function indicator feature number enables the technical activation of the
function subject to the client applying a feature activation code made available by IBM.
򐂰 The 2244 function authorization feature numbers establish the extent of IBM authorization
for that function on the 2107 machine for which it was acquired.
To use Metro Mirror you must have the 2107 function indicator feature (#0744) and the
corresponding DS8000 Series Function Authorization (IBM 2244-RMC) with the adequate
feature number (#7440-7450) to establish the extent of IBM authorization: license size in
terms of physical capacity. The DS8000 Series Function Authorizations are for billing
purposes only and establish the extent of IBM authorization for use of a particular licensed
function on the IBM System Storage DS8000 series (IBM 2107). Authorization must be
acquired for both the source and target 2107 machine.
Note: For a detailed explanation of the features involved and the considerations you must
have when ordering Metro Mirror we recommend that you refer to the announcement
letters:
򐂰 IBM System Storage DS8000 — Functions Authorization (IBM 2244).
򐂰 IBM System Storage DS8000 Series (IBM 2107)
IBM announcement letters can be found at:

http://www.ibm.com/products
Interoperability
Metro Mirror pairs can only be established between disk subsystems of the same (or similar)
type and features. For example, a DS8000 can have a Metro Mirror pair with another DS8000,
a DS6000, an ESS 800, or an ESS 750. It cannot have a Metro Mirror pair with an RVA or an
ESS F20. Note that all disk subsystems must have the appropriate Metro Mirror feature
installed. If your DS8000 is being mirrored to an ESS disk subsystem, the ESS must have
PPRC Version 2 (which supports Fibre Channel links) with the appropriate licensed internal
code level (LIC).
Refer to the DS8000 Interoperability Matrix for more information:

http://www-1.ibm.com/servers/storage/disk/ds8000/interop.html

13
Chapter 13. Metro Mirror performance and

scalability
In this chapter we discuss performance and scalability considerations that you must consider
when implementing Metro Mirror with the DS8000.

13.1 Performance
Since Metro Mirror is a synchronous mirroring technology, it introduces a performance
overhead (for write operations) over a similar environment that has no remote mirroring. On
the other side, it does not consume any CPU activity, unlike operating system mirroring. You
should understand this as part of the planning process for Metro Mirror.
Bandwidth analysis and capacity planning for your Metro Mirror links should help to define
how many links you need, and when you need to add more, to ensure best possible
performance.
As part of your implementation project, you may be able to identify and then distribute hot
spots across your configuration, or take other actions to manage and balance the load.
Some basic things you should consider:

򐂰 Is your bandwidth too little, so that you may see an increase in the response time of your
applications at moments of high workload?
򐂰 We recommend that you do not share the Metro Mirror link I/O ports with host attachment
ports. One result can be unpredictable performance of Metro Mirror and a much more
complicated search in the case of performance analysis.
򐂰 Distance is an important topic. Remember that light speed is less than 300,000 km/s, that
is, less then 300 km/ms on Fibre. The data must go to the other site, and then an
acknowledgement must come back. Add possible latency times of some active
components on the way, and you get approximately a 1-ms overhead per 100 km for write
I/Os.
򐂰 Sometimes the problem is not Metro Mirror, but rather hot spots on the disks. Be sure that
these problems are resolved before you start with Metro Mirror.
13.1.1 Initial synchronization

When doing the initial synchronization of your Metro Mirror pairs, the DS8000 uses a throttling
algorithm to prevent the Metro Mirror process from using too many source site resources.
This might prolong the synchronization process if your DS8000 is busy at the time. You can
choose to stagger your synchronization tasks or to run them at a time of low utilization to
make this process more efficient.
As an alternative, you may also choose to do the initial synchronization in the Global Copy
mode, and then switch to the Metro Mirror mode. This will allow you to bring all volumes to
duplex-pending XD state, which has little or no application impact, and then switch to full
duplex state. You can do so using the DS CLI.
13.2 Scalability
The DS8000 Metro Mirror environment can be scaled up or down as required. If new volumes
are added to the DS8000 that require mirroring, they can be dynamically added. If additional
Metro Mirror paths are required, they also can be dynamically added.
As we have previously mentioned, the logical nature of the LSS has made a Metro Mirror
implementation on the DS8000 easier to plan, implement, and manage. However, if you need
to add more LSSs to your Metro Mirror environment, your management and automation
solutions should be set up to handle this.
The eRCMF and GDPS service offerings are designed to provide this functionality. Visit the
IBM Web site and see the Services and Industry Solutions page for more information. Also
see Part 9, “Solutions” on page 537.
Adding capacity to same DS8000

If you are adding capacity to an existing DS8000, providing that your Metro Mirror link
bandwidth is not close to or over capacity, it is possible that you only have to add volume
pairs to your configuration. If you are adding more LSSs, then you must define Metro Mirror
paths before adding volume pairs.
Adding capacity in new DS8000s

If you are adding new DS8000s to your configuration, you must add physical Metro Mirror
links before defining your Metro Mirror paths and volume pairs. A minimum of two Metro
Mirror paths per DS8000 pair is recommended for redundancy reasons. Your bandwidth
analysis will indicate if you require more than two paths.
Chapter 13. Metro Mirror performance and scalability 131

14
Chapter 14. Metro Mirror interfaces and

examples
This chapter describes the interfaces that can be used for Metro Mirror management for the
IBM System Storage DS8000 in an open systems environment. In this chapter we also
present examples that illustrate step-by-step how to execute the setup and management
tasks of the Metro Mirror environment.
The following topics and examples are presented in this chapter:

򐂰 DS Command Line Interface and DS Storage Manager overview
򐂰 Set up a Metro Mirror environment
򐂰 Remove a Metro Mirror environment
򐂰 Manage the Metro Mirror environment
򐂰 Failover and Failback operations (site switch)
򐂰 freezepprc and unfreezepprc commands
򐂰 Using the DS Storage Manager GUI to manage Metro Mirror

14.1 Metro Mirror interfaces
There are various interfaces available for the configuration and management of Metro Mirror
for DS8000 when used in an open systems environment:
򐂰 Enterprise Remote Copy Facility (eRCMF)
The present chapter gives an overview of the DS CLI and the DS SM for Metro Mirror
management. DS CLI and DS SM are also covered in Part 2, “Interfaces” on page 15.

eRCMF is a scalable and flexible solution to protect business data and maintain consistent
data, providing an automation tool that simplifies the disaster recovery operations. For more
information about this solution see Part 9, “Solutions” on page 537.
Also the following interfaces, not covered in this book, can be used:
򐂰 z/OS interfaces: TSO, ICKDSF, and ANTRQST API
In order to use a z/OS interface to manage open systems LUNs, the DS8000 must have at
least one CKD volume. If you are interested in this possibility, refer to the IBM Redbook The
IBM TotalStorage DS8000 Series: Copy Services with IBM Eserver zSeries, SG24-6787.
For information about the DS Open API, refer to the publication IBM System Storage DS
Open Application Programming Interface Reference, GC35-0516.
DS CLI and DS SM similar functions for Metro Mirror management

Table 14-1 compares similar Metro Mirror commands from the DS CLI and the DS SM, and
the corresponding action.
Table 14-1 DS CLI and DS SM commands and actions

Task Command with DS CLI Select option with DS SM
Metro Mirror and Global Copy paths commands
List available I/O ports that can be lsavailpprcport This information is shown
used to establish Metro Mirror during the process when a
paths. path is established.
List established Metro Mirror lspprcpath Copy Services → Paths

paths.
Establish path. mkpprcpath Copy Services → Paths →

Create
Delete path. rmpprcpath Copy Services → Paths →

Delete
Task Command with DS CLI Select option with DS SM
Metro Mirror pairs commands
Failback. failbackpprc Copy Services → Metro

Mirror → Recover Failback
Failover. failoverpprc Copy Services → Metro

Mirror → Recover Failover
List Metro Mirror volume pairs. lspprc Copy Services → Metro

Mirror
Establish pair. mkpprc Copy Service → Metro

Mirror → Create
Suspend pair. pausepprc Copy Services → Metro

Mirror → Suspend
Resume pair. resumepprc Copy Services → Metro

Mirror → Resume
Delete pair. rmpprc Copy Services → Metro

Mirror → Delete
Freeze Consistency Group. freezepprc
Thaw Consistency Group. unfreezepprc
The DS CLI has the advantage that you can make scripts and use them for automation. The
DS Storage Manager GUI is a web-based graphical user interface and is more intuitive than
the DS CLI.
14.2 Copy Services network components

The network components and connectivity of the DS HMC are illustrated in Figure 14-1.
Copy Services Network Components
Customer
Network DS HMC 1
Ethernet Processor
(internal) Complex 0
DS Storage
Manager
Switch 1
DS Storage
Manager realtime
DS CLI DS8000
DS API
(DS HMC 2) Ethernet Processor
(external) Switch 2 Complex 1
DS Storage
Manager realtime
mark
mark
mark
Figure 14-1 DS8000 Copy Services network components
Chapter 14. Metro Mirror interfaces and examples 135

DS SM and DS CLI (as well as DS Open API) commands are issued via the Ethernet network
to the DS Hardware Management Console (DS HMC). When the DS HMC receives the
commands requests it communicates with each server in the disk subsystem via the Ethernet
network. Therefore, the DS HMC is a key component to configure and manage the DS8000
and its functions.
Each DS8000 will have an internal DS HMC in the base frame, and you can have an external
DS HMC for redundancy.
You need at least one available DS HMC to issue Copy Services commands. If you have only
one DS HMC and if it has a failure, you will not be able to issue Copy Services commands.
Therefore, we recommend having a dual DS HMC configuration, as in Figure 14-1 on
page 135, especially when using automation scripts to run Copy Services functions so that
the script keeps working in case of one DS HMC failure.
To establish a Metro Mirror pair, the LAN connection between the source and target DS8000s
is not mandatory, unlike the ESS. A similar situation happens with Global Copy. However,
copy services DS CLI commands have to be issued to the DS HMC connected to the
DS8000, which has the source volume. When you need to establish a Metro Mirror pair from
a volume at the remote site to a volume at the local site, the DS CLI command has to be
issued to the DS8000 HMC at the remote site. In this case, you need a LAN connection from
the local DS CLI client machine to the DS8000 at the remote site.
14.3 DS Command-Line Interface (DS CLI)

You can use the DS CLI interface to manage all DS8000 Copy Services functions, such as
defining paths, establishing Metro Mirror pairs, and so on. For a detailed explanation about
the DS CLI, refer to Chapter 4, “DS Command-Line Interface” on page 23.
When you establish or remove Metro Mirror paths and volume pairs, you must give the DS
CLI commands to the DS HMC that is connected to the source DS8000. Also, when checking
status information at the local and remote site, you must issue DS CLI list type commands,
such as the lspprc, to each DS8000, source, and target.
The DS CLI commands are documented in the publication IBM System Storage DS8000
Command-Line Interface User´s Guide, SC26-7916.
14.4 DS Storage Manager GUI

You can use the DS Storage Manager (DS SM) Graphical User Interface (GUI) to set up and
control Metro Mirror for DS8000 functions. It is user friendly. However, you cannot use it for
automation activities, and certain Metro Mirror functions are not supported from this interface.
For a detailed explanation of the DS SM Graphical User Interface, refer to Chapter 3, “DS
Storage Manager (GUI)” on page 17.
Note: The DS SM GUI supports (differently from the DS CLI) a multi-session environment,
but not all functions and options are supported by the GUI.
14.5 Set up Metro Mirror environment using DS CLI
In the following sections we present an example of how to set up a Metro Mirror environment
using the DS CLI. Figure 14-2 shows the configuration that we implement.
LSS10 LSS20
1000 2000
1001 2001
LSS11 Physical Fibre path

LSS21
1100 2100
1101 2101
DS8000#1 DS8000#2
-dev IBM.2107-7520781 -dev IBM.2107-75ABTV1
Figure 14-2 DS8000 configuration in the Metro Mirror setup example
In our example we use different LSS and LUN numbers for the Metro Mirror source and target
elements, so that you can more clearly understand which one is being specified when going
through the reading of the example.
Note: In a real environment (and different from our example), to simplify the management
of your Metro Mirror environment, we recommend that you maintain a symmetrical configu-
ration in terms of both physical and logical elements.
14.5.1 Preparing to work with the DS CLI

As we prepare to work with the DS CLI we do some initial tasks that will simplify our activities
during the configuration process.
Simplifying the DS CLI commands syntax

Before setting up the Metro Mirror environment, we recommend that you set up the following
DS CLI environment to simplify the commands syntax.
To save you from typing -dev storage_image_ID and -remotedev storage_image_ID in each
command, you can put these values in your CLI profile. The default and target Storage Image
ID devid and remotedevid are equivalent to the -dev storage_image_ID and -remotedev
storage_image_ID command options. See Example 14-1.
Example 14-1 Part of CLI profile
# Internal SHMC ipaddress for DS8000#1
hmc1: 10.10.10.1
# External SHMC ipaddress for DS8000#1
hmc2: 10.10.10.2
# Default and target Storage Image ID
devid: IBM.2107-7520781
remotedevid:IBM.2107-75ABTV1
For further information see Chapter 4, “DS Command-Line Interface” on page 23.

Creating password file
With the managepwfile command you can create the password file on your DS CLI client.
After creating the password file, you do not need to specify a password at login time. When
you create an operational script to manage your remote mirror and copy environment, you
can use this function so that you need not write a password in the script file. We recommend
that you use this function from a security point of view. Example 14-2 shows how to create the
password file.
Example 14-2 Creating password file

dscli> managepwfile -action add -name script1 -pw xyz1234
Date/Time: October 25, 2005 8:02:37 PM JST IBM DSCLI Version: 5.1.0.204
CMUC00206I managepwfile: Record 9.155.62.97/copy successfully added to password file
C:\Documents and Settings\Administrator\dscli\security.dat.
dscli> exit
C:\Program Files\ibm\dscli>dscli -cfg name_of_profile -user script1

Date/Time: October 25, 2005 8:03:17 PM JST IBM DSCLI Version: 5.1.0.204 DS:
IBM.2107-7520781
dscli>
14.5.2 Set up of Metro Mirror configuration

Figure 14-3 shows the configuration we set up for this example. The configuration has four
Metro Mirror pairs that reside in two LSSs. Two paths are defined between each source and
target LSS.
Source Metro Mirror paths Target

LSS10 LSS20
1000 2000
Metro Mirror Pairs
1001 2001
LSS11 Fibre Channel links

LSS21
1100 2100
Metro Mirror Pairs
1101 2101
DS8000#1 DS8000#2
Figure 14-3 Metro Mirror environment to be set up
To configure the Metro Mirror environment, we follow this procedure:

1. Determine the available Fibre Channel links for paths definition.
2. Define the paths that Metro Mirror will use.
3. Create Metro Mirror pairs.
14.5.3 Determine the available Fibre Channel links
First you must look at the available Fibre Channel links. With the lsavailpprcport command,
you can do this (see Example 14-3). You see all available port combinations between the
source and the target LSSs. You have to issue this command to the DS HMC connected to
DS8000#1, which is the Metro Mirror source.
Example 14-3 List available fibre links

dscli> lsavailpprcport -l -remotedev IBM.2107-75ABTV1 -remotewwnn 5005076303FFC663 10:20
Date/Time: October 25, 2005 9:59:28 PM JST IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-7520781
Local Port Attached Port Type Switch ID Switch Port
===================================================
I0143 I0010 FCP NA NA
FCP port ID with the lsavailpprcport command has four hexadecimal characters in the
format 0xEEAP, where EE is a port enclosure number (00–3F), A is the adapter number
(0–F), and P is the port number (0–F). The FCP port ID number is prefixed with the letter I.
You can use the -fullid parameter to display the DS8000 Storage Image ID in the command
output (see Example 14-4).
Example 14-4 List available fibre links with DS8000 Storage Image ID
dscli> lsavailpprcport -l -fullid -remotedev IBM.2107-75ABTV1 -remotewwnn 5005076303FFC663 10:20
========================================================================
IBM.2107-7520781/I0143 IBM.2107-75ABTV1/I0010 FCP NA NA
IBM.2107-7520781/I0213 IBM.2107-75ABTV1/I0140 FCP NA NA
You need the worldwide node name (WWNN) of your target DS8000 to issue the
lsavailpprcport command. You can get this by using the lssi command; see Example 14-5.
You have to issue this command to the DS HMC connected to DS8000#2, which is the Metro
Mirror target.
Example 14-5 Get WWNN of target DS8000

dscli> lssi
Date/Time: October 25, 2005 8:38:21 PM JST IBM DSCLI Version: 5.1.0.204
============================================================================
- IBM.2107-75ABTV1 IBM.2107-75ABTV0 9A2 5005076303FFC663 Online Enabled
- IBM.2107-75ABTV2 IBM.2107-75ABTV0 9A2 5005076303FFCE63 Online Enabled
14.5.4 Create Metro Mirror paths

Now you can use the mkpprcpath command to create paths between the source and target
LSSs and then verify the result with an lspprcpath command. You have to issue the
mkpprcpath command for each LSS pair; see Example 14-6.
Example 14-6 Create Metro Mirror paths and list them

dscli> mkpprcpath -remotedev IBM.2107-75ABTV1 -remotewwnn 5005076303FFC663 -srclss 10
-tgtlss 20 i0143:i0010 i0213:i0140

-tgtlss 21 i0143:i0010 i0213:i0140
dscli> lspprcpath 10-11

Src Tgt State SS Port Attached Port Tgt WWNN
=========================================================
10 20 Success FF20 I0143 I0010 5005076303FFC663
10 20 Success FF20 I0213 I0140 5005076303FFC663
11 21 Success FF21 I0143 I0010 5005076303FFC663
11 21 Success FF21 I0213 I0140 5005076303FFC663
With the lspprcpath command you can use the -fullid command flag to display the fully
qualified DS8000 Storage Image ID in the command output; see Example 14-7.
Example 14-7 List paths with DS8000 Storage Image ID

dscli> lspprcpath -fullid 10-11
===================================================================================================================
IBM.2107-7520781/10 IBM.2107-75ABTV1/20 Success FF20 IBM.2107-7520781/I0143 IBM.2107-75ABTV1/I0010 5005076303FFC663
14.5.5 Create Metro Mirror pairs

After creating the paths, you can establish Metro Mirror volume pairs. Do this by using the
mkpprc command and verifying the result with the lspprc command; see Example 14-8.
When creating a Metro Mirror pair, you must specify -type mmir parameter with the mkpprc
command.
Example 14-8 Create Metro Mirror pairs and verify the result
dscli> mkpprc -remotedev IBM.2107-75ABTV1 -type mmir 1000-1001:2000-2001 1100-1101:2100-2101
CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 1000:2000 successfully created.
dscli> lspprc 1000-1001 1100-1101
ID State Reason Type SourceLSS Timeout (secs) Critical Mode First Pass Status
===================================================================================================
1000:2000 Copy Pending - Metro Mirror 10 unknown Disabled Invalid
Once the Metro Mirror source and target volumes have been synchronized, the volume state
changes to Full Duplex from Copy Pending; see Example 14-9.
Example 14-9 List Metro Mirror status after Metro Mirror initial copy completes
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
1000:2000 Full Duplex - Metro Mirror 10 unknown Disabled Invalid
The states of full duplex and copy pending indicate the Metro Mirror source state. In the case
of the target state, the states are Target Full Duplex and Target Copy Pending; see
Example 14-10. You have to give this command to the DS HMC connected to DS8000#2,
which is the Metro Mirror target.
Example 14-10 lspprc for Metro Mirror target volumes

dscli> lspprc 2000-2001 2100-2101
Date/Time: October 26, 2005 5:28:37 PM JST IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-75ABTV1
==========================================================================================================
1000:2000 Target Copy Pending - Metro Mirror 10 unknown Disabled Invalid
dscli> lspprc 2000-2001 2100-2101
=========================================================================================================
1000:2000 Target Full Duplex - Metro Mirror 10 unknown Disabled Invalid
In the copy pending state, you can check the data transfer status of the Metro Mirror initial
copy by using the lspprc -l command. Out Of Sync Tracks shows the remaining tracks to be
sent to the target volume. The size of the logical track for FB volume for the DS8000 is 64 KB.
And you also can use the lspprc -fullid command to show the DS8000 Storage Image ID
in the command output; see Example 14-11.
Example 14-11 lspprc -l and lspprc -fullid for Metro Mirror pairs
dscli> lspprc -l 1000-1001 1100-1101
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs)
Critical Mode First Pass Status
==========================================================================================================================================
1000:2000 Copy Pending - Metro Mirror 46725 Disabled Disabled invalid - 10 unknown
Disabled Invalid
Disabled Invalid
Disabled Invalid
Disabled Invalid
dscli> lspprc -fullid 1000-1001 1100-1101
ID State Reason Type SourceLSS Timeout (secs) Critical Mode First Pass
Status
==========================================================================================================================================
IBM.2107-7520781/1000:IBM.2107-75ABTV1/2000 Full Duplex - Metro Mirror IBM.2107-7520781/10 unknown Disabled Invalid
dscli>
14.6 Remove Metro Mirror environment using DS CLI

In this section we show how to terminate the Metro Mirror environment that was set up in the
previous sections. We go through the following steps:
1. Remove Metro Mirror pairs.
2. Remove logical paths.

14.6.1 Remove Metro Mirror pairs
The rmpprc command removes the volume pairs relationships; see Example 14-12. You can
use the -quiet parameter to turn off the confirmation prompt for this command.
Example 14-12 Removing Metro Mirror pairs

dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli> rmpprc -remotedev IBM.2107-75ABTV1 1000-1001:2000-2001 1100-1101:2100-2101
CMUC00160W rmpprc: Are you sure you want to delete the Remote Mirror and Copy volume pair relationship
1000-1001:2000-2001:? [y/n]:y
CMUC00155I rmpprc: Remote Mirror and Copy volume pair 1000:2000 relationship successfully withdrawn.
1100-1101:2100-2101:? [y/n]:y
You can add the -at tgt parameter to the rmpprc command to remove only the Metro Mirror
target volume; see Example 14-13. The commands are given to the DS HMC connected to
DS8000#2, which is the Metro Mirror target.
Example 14-13 Results of rmpprc with -at tgt

dscli> lspprc 2002
=========================================================================================================
dscli>
dscli> rmpprc -quiet -remotedev IBM.2107-75ABTV1 -at tgt 1002:2002
dscli>
dscli> lspprc 2002
CMUC00234I lspprc: No Remote Mirror and Copy found.
Example 14-14 shows the Metro Mirror source volume status after the rmpprc -at tgt
command completed and the result of a rmpprc -at src command. In this case, there are still
available paths. Therefore, the source status changed after the rmpprc -at tgt command
completed. If there are no available paths, the state of the Metro Mirror source volumes is
preserved. You have to give the command to the DS HMC connected to DS8000#1, which is
the Metro Mirror source.
Example 14-14 Metro Mirror source volume status after rmpprc with -at tgt and rmpprc with -at src
dscli> lspprc 1002
==================================================================================================
<< After rmpprc -at tgt command completes >>
dscli> lspprc 1002

========================================================================================================
1002:2002 Suspended Simplex Target Metro Mirror 10 unknown Disabled Invalid
dscli> rmpprc -quiet -remotedev IBM.2107-75ABTV1 -at src 1002:2002
dscli> lspprc 1002
14.6.2 Remove paths

The rmpprcpath command removes paths. Before removing the paths, you must remove all
remote mirror pairs that are using the paths or you must use the -force parameter with the
rmpprcpath command; see Example 14-15.
Example 14-15 Remove paths

dscli> lspprc 1000-1001 1100-1101
IBM.2107-7520781
dscli>
dscli>dscli> rmpprcpath -remotedev IBM.2107-75ABTV1 10:20 11:21
IBM.2107-7520781
CMUC00152W rmpprcpath: Are you sure you want to remove the Remote Mirror and Copy path
10:20:? [y/n]:y
CMUC00150I rmpprcpath: Remote Mirror and Copy path 10:20 successfully removed.
CMUC00152W rmpprcpath: Are you sure you want to remove the Remote Mirror and Copy path
11:21:? [y/n]:y
If you do not remove the Metro Mirror pairs that are using the paths, the rmpprcpath
command fails; see Example 14-16.
Example 14-16 Removing paths still having Metro Mirror pairs

dscli> lspprc 1002
==================================================================================================
dscli>
dscli> rmpprcpath -remotedev IBM.2107-75ABTV1 -quiet 10:20
CMUN03070E rmpprcpath: 10:20: Copy Services operation failure: pairs remain

If you want to remove logical paths while still having Metro Mirror pairs, you can use the
-force parameter; see Example 14-17. After the path has been removed, the Metro Mirror
pair is still in full duplex state until the Metro Mirror source receives I/O from the servers.
When I/O arrives to the Metro Mirror source, the source volume becomes suspended. If you
set the Consistency Group option for the LSS in which the Metro Mirror volumes reside, I/Os
from the servers are held with queue full status for the specified timeout value.
Example 14-17 Removing paths while still having Metro Mirror pairs with force parameter
dscli> lspprc 1002
==================================================================================================
dscli>
dscli> rmpprcpath -remotedev IBM.2107-75ABTV1 -quiet 10:20
dscli>
dscli> rmpprcpath -remotedev IBM.2107-75ABTV1 -quiet -force 10:20
dscli> lspprc 1002
==================================================================================================
<< After I/O goes to the source volume >>
dscli> lspprc 1002

====================================================================================================================
1002:2002 Suspended Internal Conditions Target Metro Mirror 10 unknown Disabled Invalid
14.7 Manage the Metro Mirror environment with the DS CLI

In this section we show how we can manage the Metro Mirror environment —such as
suspending and resuming Metro Mirror pairs and changing logical paths.
14.7.1 Suspend and resume Metro Mirror data transfer

The pausepprc command stops Metro Mirror from transferring data to the target volumes.
After this command completes, the Metro Mirror pair becomes suspended. I/Os from the
servers complete at the Metro Mirror source volumes without sending those updates to their
target volumes; see Example 14-18.
Example 14-18 Suspending Metro Mirror data transfer

dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
dscli> pausepprc -remotedev IBM.2107-75ABTV1 1000-1001:2000-2001
CMUC00157I pausepprc: Remote Mirror and Copy volume pair 1000:2000 relationship successfully paused.
dscli>
dscli> lspprc 1000-1001 1100-1101
=======================================================================================================
1000:2000 Suspended Host Source Metro Mirror 10 unknown Disabled Invalid
Because the source DS8000 keeps track of all changed data on the source volume, you can
resume Metro Mirror operations at a later time. The resumepprc command resumes a Metro
Mirror relationship for a volume pair and restarts transferring data. You must specify the
remote mirror and copy type, such as Metro Mirror or Global Copy with the -type parameter;
see Example 14-19.
When resuming the Metro Mirror pairs, the state of the pairs is copy pending, and the way the
copy is done, data consistency at the target volumes is not kept. Therefore, you must take
specific action to keep data consistency at the recovery site while resuming Metro Mirror
pairs. Taking an initial FlashCopy at the recovery site is one of the ways to keep data
consistency.
Example 14-19 Resuming Metro Mirror pairs

dscli> lspprc 1000-1001 1100-1101
=======================================================================================================
dscli>
dscli> resumepprc -remotedev IBM.2107-75ABTV1 -type mmir 1000-1001:2000-2001
CMUC00158I resumepprc: Remote Mirror and Copy volume pair 1000:2000 relationship successfully resumed.
This message is being returned before the copy completes.
dscli>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
14.7.2 Add and remove paths

You can use the mkpprcpath command to add and reduce the number of paths associating
LSS pairs.

In Example 14-20, for each LSS pair (10:20 and 11:21), we add one additional path
(I0102:I0031) to the existing two paths (I0143:I0010, I0213:I0140).
Example 14-20 Adding paths

IBM.2107-7520781
=========================================================
10 20 Success FF20 I0143 I0010 5005076303FFC663
10 20 Success FF20 I0213 I0140 5005076303FFC663
11 21 Success FF21 I0143 I0010 5005076303FFC663
11 21 Success FF21 I0213 I0140 5005076303FFC663
dscli>
IBM.2107-7520781
===================================================
IBM.2107-7520781
===================================================
dscli>
-tgtlss 20 i0143:i0010 i0213:i0140 i0102:i0031
IBM.2107-7520781
-tgtlss 21 i0143:i0010 i0213:i0140 i0102:i0031
IBM.2107-7520781
dscli>
IBM.2107-7520781
=========================================================
10 20 Success FF20 I0143 I0010 5005076303FFC663
10 20 Success FF20 I0213 I0140 5005076303FFC663
10 20 Success FF20 I0102 I0031 5005076303FFC663
11 21 Success FF21 I0143 I0010 5005076303FFC663
11 21 Success FF21 I0213 I0140 5005076303FFC663
11 21 Success FF21 I0102 I0031 5005076303FFC663
Important: When adding paths with the mkpprcpath command you must specify all of the
paths that you want to use. This includes the existing paths. Otherwise, you lose those
definitions that were already there.
To reduce the number of paths you can also use the mkpprcpath command. In Example 14-21
for each LSS pair (10:20 and 11:21), we remove one path (I0102:I0031) from the existing
three paths (I0143:I0010, I0213:I0140,I0102:I0031).
Example 14-21 Removing paths

IBM.2107-7520781
=========================================================
10 20 Success FF20 I0143 I0010 5005076303FFC663
10 20 Success FF20 I0213 I0140 5005076303FFC663
10 20 Success FF20 I0102 I0031 5005076303FFC663
11 21 Success FF21 I0143 I0010 5005076303FFC663
11 21 Success FF21 I0213 I0140 5005076303FFC663
11 21 Success FF21 I0102 I0031 5005076303FFC663
dscli>
-tgtlss 20 i0143:i0010 i0213:i0140
IBM.2107-7520781
-tgtlss 21 i0143:i0010 i0213:i0140
IBM.2107-7520781
dscli>
IBM.2107-7520781
=========================================================
10 20 Success FF20 I0143 I0010 5005076303FFC663
10 20 Success FF20 I0213 I0140 5005076303FFC663
11 21 Success FF21 I0143 I0010 5005076303FFC663
11 21 Success FF21 I0213 I0140 5005076303FFC663
14.8 Failover and Failback functions for sites switching

In this section we describe the use of the Metro Mirror Failover and Failback functions in
planned and unplanned outage scenarios.
Planned outages
The planned outage procedures rely on two facts:
򐂰 Metro Mirror source and target volumes are in a consistent and current state.
򐂰 Both DS8000s are functional and reachable.
You can swap sites without any full copy operation by combining Metro Mirror initialization
modes.

Unplanned outages
In contrast to the assumptions for planned outages, the situation in a disaster is more difficult:
򐂰 In an unplanned outage situation, only the DS8000 at the recovery site is functioning. The
production site DS8000 may be lost or unreachable.
򐂰 In an unplanned situation, volumes at the production and recovery site may be in different
states.
򐂰 As opposed to a planned situation, where you can stop all I/Os at the production site to
make all volumes across the recovery site reach a consistent status, this cannot be done
in an unplanned situation. If not using Consistency Groups, in the case of, for example, a
power failure, you can only assume consistency at the level of a single volume pair, not at
the application level.
14.8.1 Metro Mirror Failover function

In our example, we used the Metro Mirror environment shown in Figure 14-4. We have four
Metro Mirror source volumes in DS8000#1 (serial number 7520781) at the production site
and four Metro Mirror target volumes in DS8000#2 (serial number 75ABTV1) at the recovery
site. We call the volumes at the production site A volumes, which initially are Metro Mirror
source volumes. We call the volumes at the recovery site B volumes, which initially are Metro
Mirror target volumes. During site switch operations, A and B volumes become alternatively
source and target. Therefore we add the A and B terminology for easier understanding.
Source Target
(A volumes) (B volumes)
Metro Mirror paths
LSS10 LSS20
1000 2000
Metro Mirror Pairs
1001 2001
LSS11 Fibre Channel links LSS21
1100 2100
Metro Mirror Pairs
1101 2101
DS8000#1 DS8000#2
Figure 14-4 Metro Mirror environment for sites switch example
A planned site switch using the Metro Mirror Failover function involves the following steps. If
the site switch is because of an unplanned outage, then the procedure starts from step 4 on
page 150:
1. When the planned outage window is reached, the applications at the production site (A)
must be quiesced to cease all write I/O activity. This way, there are no more updates to the
source volumes. Depending on the host operating system, it may be necessary to
dismount the source volumes.
2. Next make sure that all Metro Mirror pairs are in full duplex state. It is better to check on
both sites, on DS8000#1 and on DS8000#2. In order to do this, you must issue the lspprc
command to each DS HMC; see Example 14-22.
Example 14-22 Check Metro Mirror state at the production site and the recovery site
<< DS8000#1 >>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
<< DS8000#2 >>

dscli> lspprc 2000-2001 2100-2101
=========================================================================================================
3. You can give the commands freezepprc and unfreezepprc to ensure that no data can
possibly be transferred to the target volumes (B volumes); see Example 14-23. This
operation is optional because the application at the production site has been quiesced.
Therefore, no data is sent to the target volumes.
Example 14-23 freezepprc and unfreezepprc

dscli> freezepprc -remotedev IBM.2107-75ABTV1 10:20 11:21
IBM.2107-7520781
CMUC00161W freezepprc: Remote Mirror and Copy consistency group 10:20 successfully created.
dscli>
dscli> lspprc 1000-1001 1100-1101
================================================================================================
1000:2000 Suspended Freeze Metro Mirror 10 unknown Disabled Invalid
dscli>
dscli> unfreezepprc -remotedev IBM.2107-75ABTV1 10:20 11:21
CMUC00198I unfreezepprc: Remote Mirror and Copy pair 10:20 successfully thawed.
dscli>
dscli> lspprc 1000-1001 1100-1101
================================================================================================

Note: The freezepprc command is an LSS level command. This means that all remote
mirror and copy pairs, Metro Mirror and Global Copy, in the particular LSS will be
affected by this command. This command also removes the logical paths between the
LSS pair.
4. Give a failoverpprc command to the HMC connected with DS8000#2.

You must issue the failoverpprc command, according to the roles the volumes will have
after the command is completed. In our example, you must specify the B volumes as the
source volumes. After the failoverpprc command is successfully executed, the B
volumes become new source volumes in suspended state; see Example 14-24. The state
of the A volumes is preserved.
Note: In the case of an unplanned outage, before you issue the failoverpprc
command, you may consider disconnecting the physical links between the production
and the recovery sites. This ensures that no unexpected data transfer to the recovery
site will occur at all.
Example 14-24 failoverpprc command

dscli> lspprc 2000-2001 2100-2101
=========================================================================================================
dscli>
dscli> failoverpprc -remotedev IBM.2107-7520781 -type mmir 2000-2001:1000-1001 2100-2101:1100-1101
CMUC00196I failoverpprc: Remote Mirror and Copy pair 2000:1000 successfully reversed.
dscli>
dscli> lspprc 2000-2001 2100-2101
=====================================================================================================
Note: The lspprc command shows Target in the State column only for a target volume.
In the case of a source volume, there is no indication.
5. Create paths in the direction recovery site to production site (B → A); see Example 14-25.
You have to give the mkpprcpath command to the DS HMC connected to DS8000#2.
Although it is not strictly necessary to reverse the paths, we recommend that you do it so
that you have a well-defined situation at the end of the procedure. Additionally, you will
need the paths to transfer the updates back to the production site.
Example 14-25 Create paths from B to A

dscli> lsavailpprcport -l -remotedev IBM.2107-7520781 -remotewwnn 5005076303FFC1A5 20:10
===================================================
dscli>
===================================================
dscli>
dscli> mkpprcpath -remotedev IBM.2107-7520781 -remotewwnn 5005076303FFC1A5 -srclss 20
-tgtlss 10 i0010:i0143 i0140:i0213
dscli>
dscli> mkpprcpath -remotedev IBM.2107-7520781 -remotewwnn 5005076303FFC1A5 -srclss 21
-tgtlss 11 i0010:i0143 i0140:i0213
dscli>
=========================================================
20 10 Success FF10 I0010 I0143 5005076303FFC1A5
dscli>
6. Depending on your operating system, it may be necessary to rescan Fibre Channel

devices (to remove device objects for the recovery site volumes and recognize the new
sources) and mount the new source volumes (B volumes).
Start all applications at the recovery site (B).
Now that the applications have started, Metro Mirror starts keeping track of the updated
data on the new source volumes at B.
Summary of the failover procedure

Briefly, this is the procedure we followed to switch to the recovery site:
1. Stop applications at the production site.
2. Verify the Metro Mirror volume pairs.
3. freezepprc and unfreezepprc (optional).
4. failoverpprc B to A (to DS8000#2).
5. mkpprcpath B to A (to DS8000#2).
6. Start applications at the recovery site.

14.8.2 Metro Mirror Failback function
Once the production site has been restored, you must move your application back. At this
moment we assume that:
򐂰 Applications are updating the source volumes (B volumes) at the recovery site.
򐂰 During operation at the recovery site, data has not been replicated from B to A.
򐂰 Both DS8000s (source and target) are functional and reachable.
򐂰 Paths are already established from the recovery to the production site.
򐂰 Volumes at the recovery site are in the suspended state (source). Volumes at the
production site are also in the suspended state (source), if you executed the optional step
3 on page 151 in the previous procedure.
A switchback using the Metro Mirror Failback function involves the following steps:
1. Using the lspprcpath command, verify that the paths from the recovery site to the
production site are available. Give this command to the DS HMC connected to DS8000#2;
see Example 14-26. You can check whether you have paths between the correct DS8000s
with the -fullid parameter.
Example 14-26 Verify paths between recovery and production sites

===================================================================================================================
IBM.2107-75ABTV1/20 IBM.2107-7520781/10 Success FF10 IBM.2107-75ABTV1/I0010 IBM.2107-7520781/I0143 5005076303FFC1A5
2. If you did not reverse the paths before (in step 5 on page 151 of the previous procedure),
now you must establish paths from the recovery to the production site, before running the
failbackpprc command.
3. Run the failbackpprc command from the recovery site to the production site. You have to
give this command to the DS HMC connected to DS8000#2. The failbackpprc command
will copy all the modified tracks from the B volumes to the A volumes. See Example 14-27.
You must issue the failbackpprc command according to the roles the volumes will have
after the command is completed. In our example, you must specify the B volumes as the
source volumes and the A volumes as the target volumes. After the failbackpprc
command is successfully executed, the B volumes become source volumes in copy
pending state, and the A volumes become target volumes in target copy pending state.
Note: When issuing the failbackpprc command, if you specify the A volume as the
source and the B volume as the target, and you give the command to the DS HMC
connected to the DS8000#1, then data will be copied from A to B.
Example 14-27 failbackpprc command

<< DS8000#2 >>
Date/Time: October 28, 2005 12:22:21 AM JST IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-75ABTV1
=========================================================
dscli>
dscli> lspprc 2000-2001 2100-2101
=====================================================================================================
dscli>
dscli> failbackpprc -remotedev IBM.2107-7520781 -type mmir 2000-2001:1000-1001 2100-2101:1100-1101
CMUC00197I failbackpprc: Remote Mirror and Copy pair 2000:1000 successfully failed back.
dscli>
dscli> lspprc 2000-2001 2100-2101
===================================================================================================
<< DS8000#1 >>

dscli> lspprc 1000-1001 1100-1101
Date/Time: October 28, 2005 12:23:09 AM JST IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-7520781
==========================================================================================================
failback initialization characteristics

The failbackpprc initialization mode resynchronizes the volumes. The following apply:
򐂰 If a volume at the production site is in simplex state, all of the data for that volume is
copied back from the recovery site to the production site.
򐂰 If a volume at the production site is in the full-duplex or in suspended state and without
changed tracks, only the modified data on the volume at the recovery site is copied back
to the volume at the production site.
򐂰 If a volume at the production site is in a suspended state and has tracks on which data has
been written, then both the tracks changed on the production site and the tracks marked at
the recovery site will be copied back.
򐂰 Finally, the volume at the production site becomes a write-inhibited target volume. This
action is performed on an individual volume basis.
Notes on the failbackpprc command

If the server at the production site is still online and accessing the disk, or a crash happens,
so that the SCSI persistent reserve is still set on the source disk (A), the failbackpprc
command fails; see Example 14-28 on page 154. In this case, the server at the production
site locks the target with a SCSI persistent reserve. This situation can be reset with the
varyoffvg command (in this case on AIX), and the failbackpprc command will complete
successfully.

There is a -resetreserve parameter for the failbackpprc command. This parameter resets
the reserved status so the operation can complete. In a situation after a real disaster, you
may use this parameter because the server might go down while the SCSI persistent reserve
was set on the A volume. In a situation after a planned site switch, you must not use this
parameter because the server at the production site still owns the A volume, and may be
using it, while the Failback operation suddenly changes the contents of the volume. This may
cause the server file system’s corruption.
Example 14-28 failbackpprc command fails when A volumes are online

CMUN03103E failbackpprc: 2000:1000: Copy Services operation failure: The volume is in a long busy
state, not yet configured, not yet formatted, or the source and target volumes are of different types.
dscli>
<< After performing varyoffvg A volumes from the AIX servers at the production site >>

Now, we continue with the switchback procedure:

1. Wait until the Metro Mirror pairs become synchronized. When the pairs are synchronized,
the state of the source is full duplex and the state of the targets is target full duplex; see
Example 14-29.
Example 14-29 Confirm synchronization has been completed

<< DS8000#2 >>
dscli> lspprc 2000-2001 2100-2101
==================================================================================================
<< DS8000#1 >>

dscli> lspprc 1000-1001 1100-1101
=========================================================================================================
2. Before returning to normal operation, the applications, still updating B volumes at the
recovery site, must be quiesced to cease all write I/O from updating the B volumes.
Depending on the host operating system, it may be necessary to dismount the B volumes.
3. You should now execute one more failoverpprc command. At this time, you must specify
the A volumes as the source volumes and the B volumes as the target volumes. You must
give this command to the DS HMC connected to the DS8000#1.
This operation changes the state of the A volumes from target full duplex to (source)
suspended. The state of the B volumes is preserved; see Example 14-30.
Example 14-30 failoverpprc to convert A volumes to source suspended

<< DS8000#1 >>
dscli> lspprc 1000-1001 1100-1101
=========================================================================================================
dscli>
dscli> failoverpprc -remotedev IBM.2107-75ABTV1 -type mmir 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
dscli> lspprc 1000-1001 1100-1101
=====================================================================================================
dscli>
<< DS8000#2 >>

dscli> lspprc 2000-2001 2100-2101
==================================================================================================
4. Define paths in the direction of production site to recovery site (A → B); see
Example 14-31. You must create paths if you executed the freezepprc command in the
optional step 3 on page 151 of the production to recovery site switchover procedure
(freezepprc removed the paths).
Example 14-31 Create Metro Mirror paths from A to B

-tgtlss 20 i0143:i0010 i0213:i0140
-tgtlss 21 i0143:i0010 i0213:i0140

5. Then issue another failbackpprc command. At this time, you must specify the A volumes
as the source volumes and the B volumes as the target volumes. You must give this
command to the DS HMC connected to the DS8000#1.
After you have successfully executed the failbackpprc command, the A volumes become
source volumes in copy pending state and the B volumes become target volumes in target
copy pending state; see Example 14-32.
Example 14-32 failbackpprc command to restart A to B Metro Mirror operation

dscli> failbackpprc -remotedev IBM.2107-75ABTV1 -type mmir 1000-1001:2000-2001
1100-1101:2100-2101
6. Wait until the Metro Mirror pairs are synchronized. Normally, this operation does not take
much time because no data transfer is necessary. After the Metro Mirror pairs are
synchronized, the state of the sources volumes (A) becomes full duplex and the state of
the target volumes (B) becomes target full duplex; see Example 14-33.
Example 14-33 After the Metro Mirror pairs have been synchronized
<< DS8000#1 >>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
<< DS8000#2 >>

dscli> lspprc 2000-2001 2100-2101
=========================================================================================================
7. Depending on your operating system, it may be necessary to rescan Fibre Channel

devices and mount the new source volumes (A) at the production site.
Start all applications at the production site and check for consistency.
Now that the applications have started, all the write I/Os to the source volumes (A) are
tracked by Metro Mirror. You should verify the application’s integrity.
8. Eventually, terminate the paths from the recovery to the production LSSs.
Summary of the failback procedure

Briefly, this is the procedure we followed to switch to the production site:
1. Verify paths status B to A.
2. Eventually mkpprcpath B to A.
3. failbackpprc B to A (to DS8000#2).
4. Wait for volume pairs synchronization.
5. Quiesce applications at the recovery site (B).
6. failoverpprc A to B (to DS8000#1).
7. mkpprcpath A to B.
8. failbackpprc A to B (to DS8000#1).
9. Wait for volume pairs synchronization.
10.Start applications at the production site (A).
11.Eventually rmpprcpath B to A.
14.9 Freezepprc and unfreezepprc

As mentioned in 12.4, “Consistency Group function” on page 116, you have to implement a
certain procedure to keep data consistency at the recovery site. One of the ways is using the
freezepprc and unfreezepprc commands with the Consistency Group option. In this section,
we illustrate how to set the Consistency Group option and we discuss how the freezepprc
and unfreezepprc commands work.
You can specify the Consistency Group option when you define the Metro Mirror paths that
communicate source and target LSSs, or when you change the default Consistency Group
setting on each LSS by means of the chlss command —by default, this option is disabled.
With the chlss command you can also change the default Consistency Group timeout value,
by means of the -extlongbusy parameter.
Example 14-34 shows how to enable the Consistency Group option with the chlss command.
The setting of the option can be verified with the showlss command.
Example 14-34 Change the default Consistency Group setting with the chlss command
dscli> showlss 10
ID 10
Group 0
addrgrp 1
stgtype fb
confgvols 4
subsys 0xFF10
pprcconsistgrp Disabled
xtndlbztimout 120 secs
dscli> showlss 11
ID 11
Group 1
addrgrp 1
stgtype fb
confgvols 4
subsys 0xFF11
pprcconsistgrp Disabled
dscli>
dscli> chlss -pprcconsistgrp enable 10-11
CMUC00029I chlss: LSS 11 successfully modified.
CMUC00029I chlss: LSS 10 successfully modified.
dscli>
dscli> showlss 10
ID 10
Group 0
addrgrp 1
stgtype fb

confgvols 4
subsys 0xFF10
pprcconsistgrp Enabled
dscli>
dscli> showlss 11
ID 11
Group 1
addrgrp 1
stgtype fb
confgvols 4
subsys 0xFF11
pprcconsistgrp Enabled
When the DS8000 detects a condition where it cannot update a Metro Mirror target volume, it
notifies this condition with and SNMP alert. At that moment, if an automation procedure is in
place, the SNMP alert will trigger the automation procedure which will then execute a
freezepprc command as the one showed in Example 14-35.
Example 14-35 The results of the freezepprc command

dscli> freezepprc -remotedev IBM.2107-75ABTV1 10:20 11:21
dscli>
dscli> lspprc 1000-1001 1100-1101
================================================================================================
dscli>
CMUC00234I lspprcpath: No Remote Mirror and Copy Path found.
With the freezepprc command, the DS8000 holds the I/O activity to the addressed LSSs by
putting the source volumes in a queue full state for a time period. Example 14-36 shows, for
an AIX environment, what the iostat command reports during this time interval.
Example 14-36 AIX iostat command output report during the queue full condition
# lsvpcfg
vpath8 (Avail pv ) 75207811000 = hdisk18 (Avail ) hdisk22 (Avail )
# iostat -d vpath8 vpath9 vpath10 vpath11 1

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk18 0.0 0.0 0.0 0 0
hdisk19 100.0 0.0 0.0 0 0
hdisk21 0.0 0.0 0.0 0 0
hdisk20 0.0 0.0 0.0 0 0
hdisk25 0.0 0.0 0.0 0 0
hdisk22 100.0 0.0 0.0 0 0
hdisk23 0.0 0.0 0.0 0 0
hdisk24 0.0 0.0 0.0 0 0
Note: in addition to holding (freezing) the I/O activity, the freezepprc command also
removes the paths between the affected LSSs.
After the freezepprc for all related LSSs completes, you have consistent data at the recovery
site. Therefore, at this moment the automation procedure can execute the unfreezepprc
command to release (thaw) the I/O that was on hold (frozen) on the affected LSSs.
Example 14-37 shows the unfreezepprc command that thaws the frozen I/O queue on the
LSS pairs 10:20 and 11:21.
Example 14-37 unfreezepprc command

dscli> unfreezepprc -remotedev IBM.2107-75ABTV1 10:20 11:21
Date/Time: October 29, 2005 12:30:53 AM JST IBM DSCLI Version: 5.1.0.204 DS:
IBM.2107-7520781
In a situation where the data could not be replicated because of a links failure circumstance,
that is the production site kept running, then Metro Mirror processing can resume after the
links are recovered. Still, if the automation was triggered and the freezepprc was executed,
then the Metro Mirror paths have to be defined again. This is because the freezepprc
command removes the paths between the affected LSSs.
Then, after the paths are re-established you can execute a resumepprc command to
re-synchronize the Metro Mirror pairs. Example 14-38 shows this scenario.
Example 14-38 Resume the Metro Mirror environment

dscli> mkpprcpath -remotedev IBM.2107-75ABTV1 -remotewwnn 5005076303FFC663 -srclss 10 -tgtlss 20
i0143:i0010 i0213:i0140
dscli>
i0143:i0010 i0213:i0140
dscli>
dscli> resumepprc -remotedev IBM.2107-75ABTV1 -type mmir 1000-1001:2000-2001 1100-1101:2100-2101
CMUC00158I resumepprc: Remote Mirror and Copy volume pair 1000:2000 relationship successfully
resumed. This message is being returned before the copy completes.
dscli>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================

Note: while doing the re-synchronization, the volume pairs will be in a copy pending state.
During this time period there will be no data consistency in the target volumes. Therefore,
you may want to take specific action to keep data consistency at the recovery site while
resuming the Metro Mirror pairs. First doing a FlashCopy at the recovery site is one way to
accomplish this.
14.10 DS Storage Manager GUI examples

In this section we use the DS Storage Manager (DS SM) graphical user interface (GUI) to
manage Metro Mirror paths and pairs.
14.10.1 Create paths

This example shows you how to create Metro Mirror paths.
From the DS Storage Manager GUI you follow these steps:

3. Click Paths.
4. Select the source Storage Complex, Storage Unit, Storage Image, and LSS.
You will see the Paths panel; see Figure 14-5. This panel shows that for the selected LSS 10
at the moment there are no paths defined.
Figure 14-5 Create path - Paths panel
In the Paths panel, from the Select Action pull-down, select Create to proceed with the first
step of the creation wizard.
The creation wizard then displays the Select source LSS panel. Here you select the source
LSS; see Figure 14-6.
Figure 14-6 Create path - Step 1: select source LSS
Click Next to proceed with the second step of the wizard.
When the creation wizard displays the Select target LSS panel, you select from the pull-down
lists the storage complex, then the storage unit, then the storage image, and finally the target
LSS; see Figure 14-7.
Figure 14-7 Create path - Step 2: select target LSS
Click Next to proceed with the third step of the wizard.
When the creation wizard displays the Select source I/O ports panel, select using the check
boxes at least one I/O port (two is better) to use for Metro Mirror replication; see Figure 14-8
on page 162.

In the location column, four digits indicate the location of a port:
򐂰 The first digit (R) is to locate the frame.
򐂰 The second digit (E) is for the I/O enclosure.
򐂰 The third digit (C) is for the adapter.
򐂰 The fourth digit (P) is for the adapter’s port.
Figure 14-8 Create path - Step 3: select source I/O ports
Click Next to proceed with the fourth step of the wizard.
When the creation wizard displays the Select target I/Os ports panel, for each I/O port
selected during the third step, select from its related pull-down list the target I/O port; see
Figure 14-9.
Figure 14-9 Create path - Step 4: select target I/O ports
Click Next to proceed with the fifth step of this wizard.
When the creation wizard displays the Select path options panel, you can use the check box
to select the option Define as Consistency Group; see Figure 14-10.
See 12.4, “Consistency Group function” on page 116, for a detailed discussion of this option.
Figure 14-10 Create path - Step 5: select path options
Click Next to proceed with the sixth and last step of this wizard.
Figure 14-11 shows the Verification panel. Here you check all the components of your path
configuration and, if necessary, click Back to correct any of them or click Finish to validate
the configuration and end the wizard.
Figure 14-11 Create path - Step 6: verification

After you finish you are again presented the Paths panel, and now you will see the existing
paths that you have just created; see Figure 14-12.
Figure 14-12 Create path - result
14.10.2 Add paths

There are situations where you may want to add additional paths to the existing ones, for an
LSS pair. For example:
򐂰 To add redundant paths.
򐂰 To increase available bandwidth.
򐂰 To change the ports you are using for Metro Mirror. With the GUI you must first add the
new paths before you delete old paths.
We will see now an example of how to add a path to an LSS pair which already has a path
defined. From the DS Storage Manager GUI you follow these steps:
3. Click Paths.
You will see the Paths panel; see Figure 14-13. This panel shows one existing path for the
LSS pair 11:10.
Figure 14-13 Add paths - Paths panel
In the Paths panel, from the Select Action pull-down, select Create to proceed with the first
step of the creation wizard.
The creation wizard then displays the Select source LSS panel. Here we select the source
LSS 11; see Figure 14-14.
Figure 14-14 Add paths - Step 1: select source LSS
Click Next to proceed with the second step of the wizard.

When the creation wizard displays the Select target LSS panel, you select from the pull-down
lists the storage complex, then the storage unit, then the storage image, and finally the target
LSS 10; see Figure 14-15.
Figure 14-15 Add paths - Step 2: select target LSS
Click Next to proceed with the third step of the wizard.
When the creation wizard displays the Select source I/O ports panel, select using the check
boxes the port on the source for the additional path that you are defining; see Figure 14-16.
Figure 14-16 Add paths - Step 3: select source I/O ports
Click Next to proceed with the fourth step of the wizard.
When the creation wizard displays the Select target I/Os ports panel, you select from the
pull-down list the target I/O port; see Figure 14-17.
Figure 14-17 Add paths - Step 4: select target I/O port
When the creation wizard displays the Select path options panel, you can use the check box
to select the option Define as Consistency Group; see Figure 14-18.
See 12.4, “Consistency Group function” on page 116, for a detailed discussion of this option.
Figure 14-18 Add paths - Step 5: select path options

Figure 14-19 shows the Verification panel. Here you check all the components of your paths
Figure 14-19 Add paths - Step 6: verification
After you finish you are again presented the Paths panel, and now you will see the new path
recently added; see Figure 14-20.
Figure 14-20 Add paths - Result
Note: In our example, the defined paths are in one direction. If you want to mirror from the
target back to the source you will have to establish paths in the other direction.
14.10.3 Change options

You can change options of the existing paths. For this, from the DS Storage Manager GUI you
follow these steps:
3. Click Paths.
You will see the Paths panel; see Figure 14-21. This panel shows a list of the existing paths
for the selected LSS 11. In this panel we check the box for the path we want to work with and
from the Select Action pull-down list we select LSS copy options.
Figure 14-21 Path panel - LSS copy options
The LSS Copy Options panel is then displayed; see Figure 14-22. Here you can change the
options of the defined path: various timeout values, as well as the critical mode attribute. For
a discussion of these options see Chapter 12, “Metro Mirror options and configuration” on
page 113.
Figure 14-22 LSS copy options panel
When finished click OK.
14.10.4 Delete paths

In this example we show you how to remove a path. From the DS Storage Manager GUI you
follow these steps:
3. Click Paths.

You will see the Paths panel. This panel shows the list of established paths for the selected
LSS 11 of our example. Here you check the box to select the paths you want to remove; see
Figure 14-23.
Then from the Select Action pull-down list you select Delete.
Figure 14-23 Delete paths - Paths panel
A warning pops up; see Figure 14-24.
Figure 14-24 Delete paths warning
Note: Deleting paths reduces bandwidth. For a discussion on this topic see the
Chapter 13, “Metro Mirror performance and scalability” on page 129.
After you click Continue the deletion is completed. Then you are presented again the Paths
panel where you will see the path is not anymore there; see Figure 14-25. If the path still
appears to be in the list, click Refresh to renew the screen.
Figure 14-25 Delete paths - Result
14.10.5 Create volume pairs

In this example we are going to see how the Metro Mirror volume pairs are created.
Some things to remember when you are going to create the volume pairs, are the following:
򐂰 All volumes, on the source and also on the target, must exist before starting to create the
Metro Mirror pairs.
򐂰 Source and target volumes should be of the same size. The target volume can be bigger
than the source volume, but if this is the case, you cannot reverse the direction of the pair.
So different sizes of source and target volumes makes sense only for migration purposes.
򐂰 Usually before you start creating the Metro Mirror pairs, you have to define the paths
between the source and target LSS, as explained in “Create paths” on page 160.
To start creating volume pairs, from the DS Storage Manager GUI you follow these steps:
3. Click Metro Mirror.

4. The Metro Mirror panel is displayed; see Figure 14-26. Here you select the source Storage
Complex, Storage Unit, Storage Image, and Resource Type. Possible Resource Type
options are:
– Lss, from which you select the LSS number
– Host attachment, from which you select the host attachment name
– Volume Group, from which you select the Volume Group name
– Show All Volumes, from which you select All FB volumes or All CKD volumes
Figure 14-26 Create MM pair - Metro Mirror panel
In the Metro Mirror panel, from the Select Action pull-down menu you select Create to
proceed with the first step of the creation wizard.
The Create Metro Mirror panel is displayed; see Figure 14-27. Here you select the volume
pairing method. If you select Automated volume pair assignment the system will later
automatically select target volumes of the same size as the source volume. If you select
Manual volume pair assignment you have to manually assign to each source volume a
target volume, one after the other. In our example we selected Automated volume pair
assignment.
Figure 14-27 Create MM pair - Step 1: volume pairing method
Click Next to proceed with the second step of the creation wizard.
The Select source volumes panel is displayed; see Figure 14-28. Here you can check the
boxes to select the source volumes of the Metro Mirror pairs. If you still have not established
paths between the source and target LSSs, you can do this now by clicking the Create Paths
button in this panel and proceeding as explained in 14.10.1, “Create paths” on page 160.
Figure 14-28 Create MM pair - Step 2: select source volumes
Click Next to proceed with the third step of the creation wizard.
The Select target volumes (Auto pairing) panel is displayed; see Figure 14-29. Here you can
select all target volumes from the available ones on the target LSS.
Figure 14-29 Create MM pair - Step 3: select target volumes (Auto pairing)
Click Next to proceed with the fourth step of the creation wizard.

The Select copy options panel is displayed; see Figure 14-30. Here you can select the copy
options — we select Metro Mirror. For a discussion of the options presented in this panel see
Chapter 12, “Metro Mirror options and configuration” on page 113.
Figure 14-30 Create MM pair - Step 4: select copy options
Click Next to proceed with the fifth step of the creation wizard.
The Verification panel is displayed; see Figure 14-31. Here, you can verify the Metro Mirror
volume pairs just created. Click Finish to execute this configuration.
Figure 14-31 Create MM pair - Step 5: verification
The Metro Mirror panel will be presented again; see Figure 14-32 on page 175. This time it
will show the list of volume pairs just created. You may see that the state of the volumes is
Copy pending. This indicates that the initial copy from the source to the target volumes is still
in progress. Click Refresh to see if the status has changed.
After a period of time, depending on the size and number of volumes, all volumes will be
synchronized (in-sync). At that moment the State column of the Metro Mirror panel will show
Full duplex for the volumes, as illustrated in Figure 14-32.
Figure 14-32 Create MM pair - Volume pairs created and in full duplex state
Note: On very high workloads, a high number of established Metro Mirror volumes, or
sharing of ports for Metro Mirror and host attachments, you will possibly have higher
response times for your applications during this synchronization time.
14.10.6 Suspend volume pairs

To get access to the target volumes or to do maintenance of the target disk subsystem you
may need to suspend the Metro Mirror pairs. When the volumes are suspended the updates
on the volumes are logged, so when you after have to resume the mirroring process only the
differences will be replicated between the volumes —this is the re-synchronization process.
To suspend volume pairs, from the DS Storage Manager GUI you follow these steps:

4. The Metro Mirror panel is displayed: see Figure 14-33. Here you select the source Storage
Complex, Storage Unit, Storage Image, and Resource Type. Possible Resource Type
options are:
5. Check the boxes for the selected volume pairs you want to suspend.
6. Then, from the Select Action pull-down menu select Suspend.
Figure 14-33 Suspend - Metro Mirror panel
7. The Suspend Metro Mirror panel is displayed; see Figure 14-34. Here you are prompted to
suspend the pair at either the source or the target. It is more usual to suspend at the
source, as shown in our example. Then click OK.
Figure 14-34 Suspend - Suspend at source or target
8. Next the Metro Mirror panel is displayed, now showing the selected pairs in suspended
mode; see Figure 14-35.
Figure 14-35 Suspend - Result
14.10.7 Resume volume pairs

For the suspended volumes to reach again the full duplex state, you need to resume the
Metro Mirror volume pairs. Before starting, verify that there are paths available between the
corresponding LSSs.
To resume volume pairs, from the DS Storage Manager GUI you follow these steps:
4. The Metro Mirror panel is displayed; see Figure 14-36 on page 178. Here you select the
source Storage Complex, Storage Unit, Storage Image, and Resource Type. Possible
Resource Type options are:

5. Check the boxes for the selected volume pairs you want to resume. In Figure 14-36 you
can see their current state is suspended.
6. Then, from the Select Action pull-down menu select Resume.
Figure 14-36 Resume - Metro Mirror panel
7. The Resume Metro Mirror panel is displayed; see Figure 14-37. Here you can now confirm
your choices. Click OK to proceed.
Figure 14-37 Resume - Verify
8. Next the Metro Mirror panel is displayed, now showing the selected volumes in full duplex
mode after the re-synchronization process is completed; see Figure 14-38. You may need
to refresh the screen to see when the volumes reach the full duplex state.
Figure 14-38 Resume - Result
14.10.8 Metro Mirror Failover
In the following example, we have two disk subsystems, a DS8000 (machine type 2107) and a
DS6000 (machine type 1750). Each disk subsystem has two volumes. The DS8000 is the
production unit. The DS6000 is the backup unit. See Table 14-2.
Table 14-2 Failover Failback example devices

Machine serial Location Normal state Volume 1 Volume 2
2105- 7503461 Production site (A) Source 1405 1406
1750-1300247 Backup site (B) Target 1805 1806
Now we will use these two machines to describe a sites switch scenario.
Site switch - production site to recovery site

The Metro Mirror Failover function changes the target volume (B) to suspended source
volume, while leaving the original source volume (A) in its current state. This command
succeeds even if the paths are down and the volume (A) at the production site is unavailable
or nonexistent. You can then access the former target volume (B). The assumption here is
that you are now running production on what was the original target disk subsystem (B). This
means that changes to the volumes on the recovery site disk subsystem will later need to be
copied back to the production site disk subsystem (A).
Note: If you try to access this volume on the same server where the previous source was
or still is, some operating systems have some problems with disks with the same serial
number or signature. In most cases, there is a procedure to do this; see Appendix A,
“Open systems specifics” on page 591.
In this example, we assume that the production site has failed. We decide to start production
processing using the backup servers at the recovery site. We need to start using the Metro
Mirror target volumes (B). These volumes must become source volumes since we will be
writing them, and these changes must eventually be mirrored back to the production site. The
procedure we follow is:
4. Select the target Storage complex, Storage unit, Storage image, and Resource Type.
Possible Resource Type options are:
– LSS from which you select the LSS number.
– Host attachment from which you select the host attachment name.
– Volume Group from which you select the Volume Group name.
– Show All Volumes from which you select All FB volumes or All CKD volumes
5. Select the volume pairs that you want to fail over.

6. Now select Recover Failover from the Select Action pull-down menu, as shown in
Figure 14-39. Take note that in this example we display the target machine (serial
13-00247) and select the target volumes. We do not work with the source machine.
Figure 14-39 Failover - Metro Mirror panel
7. We now verify the volumes with which we are working. When ready, click OK to proceed,
as shown in Figure 14-40.
Figure 14-40 Failover verify
8. The failover operation now takes place. We can see the result in Figure 14-41. Note that
Figure 14-41 shows the status of the B volumes, which are suspended. The A volumes will
still be in full duplex. This is because the Metro Mirror Failover function does not care
about the state of the former source volumes (A). We are now able to start the servers at
the backup site, and they can access the B volumes on the backup site DS6000.
Figure 14-41 Result of a failover
14.10.9 Metro Mirror Failback
We run our production using servers at the recovery site, which access our backup disk
subsystem (B). We now get ready to switch processing back to the production site (A). The
first thing to do is mirror the changes back from the backup site (B) to the production site (A).
We do this using the Metro Mirror Failback function.
Site switch - recovery site to production site

Metro Mirror Failback synchronizes the volumes at the production site (A) with the volumes at
the recovery site (B). The (B) volumes were updated while production processing was
temporarily run at the recovery site. The steps are:
4. Select the source Storage complex, Storage unit, Storage image, and Resource Type.
Possible Resource Type options are:
– LSS from which you select the LSS number.
– Host attachment from which you select the host attachment name.
– Volume Group from which you select the Volume Group name.
– Show All Volumes from which you select All FB volumes or All CKD volumes
5. Select the volume pairs you want to fail back.
6. Now select Recover Failback from the Select Action pull-down menu, as shown in
Figure 14-42. Note that in this example, we are still working with the Storage Unit at the
backup site (serial 13-00247).
Figure 14-42 Failback - Metro Mirror panel

7. We now verify the volumes we have selected; see Figure 14-43. We can suspend the pair
after the synchronization and reset the LUN reservation. Depending on whether or not
data changes are taking place on the previous source volume, the failback command will
operate; see 12.3, “Failover and Failback” on page 115. Click OK to proceed.
Figure 14-43 Failback - Verify
During the failback, changes are copied. You can click Refresh to see if the copy is
finished. When the copy is complete, the volumes at the backup site (B) will show as full
duplex source volumes, as depicted in Figure 14-44.
Figure 14-44 Failback - Result
Note: If the failback fails, check whether your prior active server can still access the
new target volume, or has not reset the reserve on the volume, and if your Metro Mirror
path is established between these LSSs.
If we display the status of the volumes at the production site (on serial 7503461), they will
appear as full duplex targets, as shown in Figure 14-45. The source machine is serial
13-00247 at the backup site.
Figure 14-45 Failback - View from the production site Storage Unit
8. At this point, we shut down the servers at the backup site. Once they are shutdown, we
know that no more changes are written to the volumes at the backup site. Now we repeat
the failover and failback process, but we do so from the production site storage unit (A).
In Figure 14-46, we select the volumes on the production storage unit and perform a
Failover.
Figure 14-46 Starting failover - View on production site DS8000

9. When the Failover is complete, the volumes on the production site show as suspended
Metro Mirror source volumes. Now, perform a Failback using the production site storage
unit, as shown in Figure 14-47.
Figure 14-47 Starting failback on the production site DS8000
When the final Failback is complete, the production site storage unit is now the source again
and the backup site storage unit is now the target again. This means the production site
storage unit (in our example, serial 75-03461) now looks like Figure 14-48. This shows serial
75-03461 as the source machine while serial 13-00247 is the target machine. If we instead
take the view from the backup site storage unit (in our example, serial 13-00247), it now looks
like it did in Figure 14-39 on page 180, prior to starting this whole exercise.
Figure 14-48 Production site storage unit at conclusion of disaster recovery exercise
Note: If the Failback fails, check whether your prior active server can still access the new
target volume, or has not reset the reserve on the volume, and if your Metro Mirror path is
established between these LSSs. You might need to use the Reset Reservation option
depicted in Figure 14-43.
Part 5
Part 5 Global Copy

In this part of the book, we describe IBM System Storage Global Copy. After presenting an
overview of Global Copy, we discuss the options available, the interfaces you can use, and
the configuration considerations. We also provide examples of the use of Global Copy.

15
Chapter 15. Global Copy overview

In this chapter we describe the characteristics and operation of Global Copy. Also discussed
are the considerations for its implementation with the IBM System Storage DS8000.

15.1 Global Copy overview
Global Copy (formerly know as PPRC Extended Distance, PPRC-XD) is a non-synchronous
remote copy function for open systems and System z environments, for longer distances than
are possible with Metro Mirror. It is appropriate for remote data migration, off-site backups,
and transmission of inactive database logs at virtually unlimited distances.
2 Write acknowledge
Server
write
1 3
Primary Secondary
(source) (target)
4
LUN or Write to secondary LUN or
volume (non-synchronously) volume
Figure 15-1 Global Copy
With Global Copy, write operations complete on the source disk subsystem before they are
received by the target disk subsystem. This capability is designed to prevent the local
system’s performance from being affected by wait time from writes on the remote system.
Therefore, the source and target copies can be separated by any distance. Figure 15-1
illustrates how Global Copy operates:
1. The host server makes a write I/O to the source (local) DS8000. The write is staged
through cache and non-volatile storage (NVS).
2. The write returns as completed to the host server’s application.
3. At a later time, that is, in a non-synchronous manner, the source DS8000 sends the
necessary data so that the updates are reflected on the target (remote) volumes. The
updates are grouped in batches for efficient transmission.
4. The target DS8000 returns write complete to the source DS8000 when the updates are
secured in the target DS8000 cache and NVS. The source DS8000 then resets its Global
Copy change recording information.
Note: The efficient extended distance mirroring technique of Global Copy is achieved with
sophisticated algorithms. For example, if changed data is in the cache, then Global Copy
sends only the changed sectors. There are also sophisticated queuing algorithms to
schedule the processing of updated the tracks for each volume and set the batches of
updates to be transmitted.
15.2 Volume states and change logic

Figure 15-2 illustrates the basic states and the change logic of a volume that is in either a
Metro Mirror or a Global Copy relationship. The following considerations apply to the volume
states when the pair is a Global Copy pair:
򐂰 Simplex: The volume is not in a Global Copy relationship.
򐂰 Suspended: In this state, the writes to the source volume are not mirrored onto the target
volume. The target volume becomes out-of-sync. During this time, Global Copy keeps a
bitmap record of the changed tracks in the source volume. Later, the volume pair can be
re-started, and then only the tracks that were updated are copied.
򐂰 Full duplex: A Global Copy volume never reaches this volume state. In the full duplex
condition updates on the source volumes are synchronously mirrored to the target
volume. With Metro Mirror this is possible. With Global Copy, even when no tracks are
out-of-sync, still the pair remains in copy pending status.
򐂰 Copy pending: Updates on the source volume are non-synchronously mirrored to the
target volume.
Simplex
Terminate Terminate
Establish Metro
Establish Global Mirror
Copy
Go to sync
Copy Pending Go to sync and

Full Duplex
suspend
Resync Terminate
Resync
Suspend Suspend
Suspened
Figure 15-2 Global Copy and Mirror state change logic
In regard to the state change logic, the following considerations apply:

򐂰 When you initially establish a mirror relationship from a volume in simplex state, you have
the option to request that it becomes a Global Copy pair (establish Global Copy arrow in
Chapter 15. Global Copy overview 189

Figure 15-2 on page 189) or a Metro Mirror pair (establish Metro Mirror arrow in
Figure 15-2 on page 189).
򐂰 Pairs can change from the copy pending state to the full duplex state when a go-to-SYNC
operation is executed (go-to-SYNC arrow in Figure 15-2 on page 189).
򐂰 You can also request that a pair be suspended as soon as it reaches the full-duplex state
(go-to-SYNC and suspend in Figure 15-2 on page 189).
򐂰 Pairs cannot change directly from full duplex state to copy pending state. They must go
through an intermediate suspend state.
򐂰 You can go from suspended state to copy pending state during an incremental copy
(copying out-of-sync tracks only). This process is similar to the traditional transition from
suspended state to full duplex state (Resync arrow in Figure 15-2 on page 189).
15.3 Global Copy positioning

Figure 15-3 lists the main points for considering Global Copy.
Global
Global Copy
Copy is
is aa recommended
recommended solution
solution for
for remote
remote data
data copy,
copy, data
data
migration
migration and
and offsite
offsite backup
backup -- over
over continental
continental distances
distances without
without
impacting
impacting application
application performance.
performance.
ItIt can
can be
be used
used for
for application
application recovery
recovery implementations
implementations ifif application
application I/O
I/O
activity
activity can
can be
be quiesced
quiesced and
and non-zero
non-zero data
data loss
loss RPO
RPO is
is admissible.
admissible.
It can be used over continental distances with excellent application performance

The distances only limited by the network and channel extenders capabilities
Application write operations do not have synchronous-like overheads
Fuzzy copy of data at the recovery site (sequence of dependent writes may not be
respected at the recovery site)
Recovery data can become a consistent point-in-time copy of the primary data, if
appropiate application checkpoints are set to do global catch-ups
Pairs are synchronized with application group consistency
Synchronizations can be done more frequently, because of short catch-ups
RPO still not zero, but improves substantially
Figure 15-3 Global Copy positioning
As summarized in Figure 15-3, Global Copy is a recommended solution when you want to
perform remote data copy, data migration, off-site backup, and transmission of inactive
database logs without impacting application performance, which is particularly relevant when
implemented over continental distances. Global Copy can also be used for application
recovery solutions based on periodic point-in-time copies of the data. This requires short
quiescings of the application’s I/O activity.
16
Chapter 16. Global Copy options and

configuration
This chapter discusses the options available when using Global Copy. It also discusses the
configuration guidelines that should be considered when planning for Global Copy with the
IBM System Storage DS8000.

16.1 Global Copy basic options
First we review the basic options available when working with Global Copy. You will see that
many of these options are common to Metro Mirror. You still must consider that the results
may differ, as Metro Mirror and Global Copy have differences in the way they work.
16.1.1 Establish Global Copy pair

This is the operation where you establish a Global Copy relationship between a source
volume and a target volume — that is, you establish a Global Copy pair. During this operation
the volumes will transition from the simplex state to the copy pending state.
When you establish a Global Copy pair, the following options are available:
򐂰 No copy
This option does not copy data from the source to the target volume. This option assumes
that the volumes are already synchronized. The data synchronization is your responsibility
and the DS8000 does not check its synchronization.
򐂰 Target read
This option allows host servers to read from the target volume. For a host server to read
the volume, the volume pair must be in full duplex state. This parameter applies to open
systems volumes. It does not apply to System z volumes.
In the open system’s file system environment, even if an application reads data from a file
system, a SCSI write command may be issued to the LUN in which the file system resides,
because some information, such as the last access time stamp may be updated by the file
system. In this case, even if you specify this option, the operation may fail.
򐂰 Suspend after data synchronization
This option suspends the remote copy pairs after the data has been synchronized. You
can use this option with the Go-to-sync operation (the Go-to-sync operation is discussed
later in 16.1.5, “Converting a Global Copy pair to Metro Mirror” on page 193.
򐂰 Wait option
This option delays the command response until the volumes are in one of the final states:
simplex, full duplex, suspended, target full duplex, target suspended. This parameter
cannot be used with -type gcp or -mode nocp, but it can be used with the Go-to-sync
operation.
򐂰 Reset reserve on target
This option allows the establishment of a Global Copy relationship when the target volume
is reserved by another host. If this option is not specified and the target volume is
reserved, the command fails. This parameter can only be used with open system volumes.
16.1.2 Suspend Global Copy Pair

This operation stops copying data to the target volume and leaves the pair in suspended
state. Because the source DS8000 keeps record of all changed tracks on the source volume,
you can resume the remote copy operation at a later time.
16.1.3 Resume Global Copy Pair

This operation resumes a Global Copy relationship for a volume pair and starts to copy data
back again to the target volume. Only modified tracks are sent to the target volume because
the DS8000 kept a record of all changed tracks on the source volume while the volumes were
suspended. When resuming a Global Copy relationship, you can use the same options you
use to initially establish a Global Copy pair, except for the no copy option.
16.1.4 Terminate Global Copy Pair

This operation ends the remote copy relationship between the volume pair. The volumes
return to the simplex state.
16.1.5 Converting a Global Copy pair to Metro Mirror

This operation is known as the Go-to-sync operation. There are two common situations in
which you would convert a pair from Global Copy mode to Metro Mirror mode:
򐂰 Situation 1: You have used Global Copy to complete the bulk transfer of data in the
creation of many copy pairs, and you now want to convert some or all of those pairs to
Metro Mirror mode.
򐂰 Situation 2: You have Global Copy pairs for which you want to make FlashCopy backups
on the recovery site. You convert the pairs temporarily to Metro Mirror mode in order to
obtain point-in-time consistent copies.
You can convert a Global Copy pair to a Metro Mirror pair using the DS CLI or DS GUI.
16.2 Creating a consistent point-in-time copy

While the volume pairs are in the copy pending state, the target volumes maintain a fuzzy
copy of the data:
򐂰 Because of the non-synchronous data transfer characteristics, at any time there will be a
certain amount of updated data that is not reflected at the target volume. This data
corresponds to the sectors that were updated since the last volume bitmap scan was
done. These are out-of-sync sectors.
򐂰 Because of the bitmap scan method, writes are not ensured to be applied onto the target
volume in the same sequence that they are written to the source volume.
When terminating the Global Copy relationship to establish host access to the target volumes,
the first issue may cause loss of transactions. Since a file system’s or database’s consistency
depends on the correct ordering of write sequences, the second issue can cause inconsistent
volumes. Therefore, to use target volumes by the host systems, you must make them
point-in-time consistent:
򐂰 The application must be quiesced and the volume pairs temporarily suspended. This is
necessary for ensuring consistency not only at the volume level, but also at the application
level.
򐂰 The target volumes have to catch up to their source counterparts. Global Copy catch-up is
the name of the transition that occurs to a Global Copy pair when it goes from its normal
out-of-sync condition until it reaches a full sync condition. At the end of this transition, the
source and target volumes become fully synchronized.
򐂰 You should now perform a FlashCopy of the target volumes onto tertiary volumes, and
then resume the Global Copy pairs. These tertiary volumes are then a consistent
point-in-time copy of the primary volumes.
Chapter 16. Global Copy options and configuration 193

16.2.1 Procedure to take a consistent point-in-time copy
Figure 16-1 summarizes the procedure, which is a typical procedure to provide a consistent
point-in-time copy of the data. In Figure 16-1 we call the Global Copy source volume at the
production site the A volume, the Global Copy target volume at the recovery site the B
volume, and the FlashCopy target volume at the recovery site the C volume.
Application running
1.Updated data is being sent

in normal Global Copy
source target
A volume B volume
C volume
Quiesce Application
2.Quiesce Application
source target
3.Go-to-sync and suspend

volume pairs
source target
Restart Application
4.Restart Application
source target Now we have
consistent
Application running data
5.Take FlashCopy B to C. FlashCopy
Return to 1.
source target
Production site Recovery site

Figure 16-1 Create consistent data in Global Copy
The steps in the procedure are the following:

1. Normal Global Copy operation.
In this phase, Global Copy runs normally. Data written to the source volume (A) is copied
to the target volume (B) independently of the write sequence to the source volume.
Therefore, at this phase, there is no guarantee that the target volume (B) holds a
consistent copy of the source volume (A) data.
2. Quiesce the application at the production site.
When you want to have a consistent copy at the recovery site, the application must be
quiesced to cease all write I/O from updating the source volume (A). Depending on the
host operating system, it may be necessary to dismount the source volume.
Note: In a DB2® environment, you can use a function provided by DB2 to stop the
application write I/O, which is the set write suspend/resume command.
3. Go-to-sync and suspend the pairs.

Perform the catch-up by issuing a Go-to-sync operation. The volume pair will leave the
copy pending state and it will reach the full duplex state. Wait for the synchronization of the
pair and suspend the pair after the synchronization has completed. You can use the single
mkpprc -type mmir -suspend command to perform a Go-to-sync and suspend operation.
After this step, the Global Copy target (B) will be holding a consistent copy of the data.
4. Restart the application at the production site.
Now that you have a consistent copy of the data at the recovery site, while no updates to
the source (A) volumes are sent due to the suspend state, you can restart the application
at the production site.
5. Take a FlashCopy B to C.
In order to resume the Global Copy operation, we have to first perform a FlashCopy from
the Global Copy target volumes (B) to the FlashCopy target volumes (C) to keep
consistent data at the recovery site. Otherwise, further Global Copy data transmissions
will update the Global Copy targets (B), modifying the consistent copy that at the moment
we have at the recovery site.
You can use the Remote FlashCopy function provided by the DS8000 to issue the
FlashCopy command to the DS8000 at the recovery site via the Global Copy paths. (If you
use Remote FlashCopy, you do not need to set up a LAN connection to the remote
DS8000.) After taking the FlashCopy, you have a consistent copy in the FlashCopy target
volume (C). You can then resume the Global Copy normal operation.
16.2.2 Making a FlashCopy at the remote site

In step 5 of the previous procedure, when doing a remote FlashCopy from the production to
the recovery site, you must take into account that a disaster may occur while the FlashCopy
establishment for multiple volumes is being taken; see Figure 16-2.
FlashCopy
1.Establish FlashCopy GC target

FC target Vol 1
FlashCopy

FC target Vol 2

Vol 3

Vol 4
DS CLI client
Recovery site DS8000
Figure 16-2 FlashCopy establishment from the production site
In Figure 16-2, four FlashCopy pairs are established at the recovery site. The FlashCopy
targets are Vol 1, Vol 2, Vol 3, and Vol 4. In this case, the DS CLI client issuing the FlashCopy
establish command is in the production site. When the DS CLI client is giving the FlashCopy
establish command, a disaster may occur. It may happen, although it is a very short window,
that the DS CLI client can issue the first and second FlashCopy establish commands before

going down. If this scenario occurs, the four volumes at the recovery site are not at the same
step of the point-in-time copy. In other words, Vol 1 and Vol 2 have the latest copies, but Vol 3
and Vol 4 have previous copies. If you detect this situation, depending on your Global Copy
operational scenario, you may be able to use the Global Copy target volumes, which may still
have consistent point-in-time copies.
As long as you issue the FlashCopy commands from the production site, the above situation
can occur when you use any FlashCopy establish command, such as the mkflash,
resyncflash, mkremoteflash, and resyncremoteflash commands.
In order to avoid or detect this situation, you can implement several procedures, which
include:
򐂰 Have the DS CLI client at the recovery site issue the FlashCopy command to the DS8000
in the recovery site. We also recommend that you keep the DS CLI execution log at the
recovery site to determine what processes have been performed during a disaster.
򐂰 If you prefer having the DS CLI client at the production site, keep the DS CLI execution log
at the recovery site to determine what processes have been performed during a disaster.
򐂰 If you use the Incremental FlashCopy function, you can use the FlashCopy sequence
number. When you establish FlashCopy, you can add a certain FlashCopy sequence
number to the set of the FlashCopy pairs. This sequence number can be used as an
identifier for a FlashCopy relation or group of FlashCopy relations. When a disaster
occurs, you can find whether you have the same set of the FlashCopy or not from this
sequence number. However, for safer implementation, we also recommend that you keep
the DS CLI execution log at the recovery site to determine what processes have been
performed during a disaster.
16.3 Hardware requirements

Global Copy is an optional licensed function of the IBM System Storage DS8000. Licensed
functions require the selection of a DS8000 series feature number (IBM 2107) and the
acquisition of DS8000 series Function Authorization (IBM 2244) feature numbers.
Authorization must be acquired for both the source and target 2107 machine.
You can get Global Copy acquiring either the Metro Mirror licensed function or the Global
Mirror licensed function. See 12.11, “Hardware requirements” on page 127, for more
information.
Interoperability
Global Copy pairs can only be established between disk subsystems of the same (or similar)
type and features. For example, a DS8000 can have a Global Copy pair with another DS8000,
a DS6000, an ESS 800, or an ESS 750. It cannot have a Global Copy pair with an RVA or an
ESS F20. Note that all disk subsystems must have the appropriate Global Copy feature
installed. If your DS8000 is being mirrored to an ESS disk subsystem, the ESS must have
PPRC Version 2 (which supports Fibre Channel links) with the appropriate licensed internal
code level (LIC).
Refer to the DS8000 Interoperability Matrix for more information:

16.4 Global Copy connectivity - paths and links
Global Copy pairs are set up between volumes in LSSs, usually in different disk subsystems,
and these are normally in separate locations. A path (or group of paths) needs to be defined
between the source LSS and the target LSS. These logical paths are defined over physical
links between the disk subsystems.
When we define paths we do not specify any particular remote copy function — paths can be
used for Global Copy, Metro Mirror, or Global Mirror replication. Therefore, the discussion
presented in 12.5, “Metro Mirror paths and links” on page 121, is also valid for Global Copy.
We recommend its reading.
16.4.1 Global Copy Fibre Channel links

The discussion presented in 12.5.1, “Fibre Channel links” on page 122, is also valid for Global
Copy. We recommend its reading.
16.4.2 Logical paths

The discussion presented in 12.5.1, “Fibre Channel links” on page 122, is also valid for Global
Copy. We recommend its reading.
16.5 Bandwidth
To determine the bandwidth requirement for your Global Copy environment, you should
consider both the amount of data you need to transfer to the remote site and the available
window.
If using Global Copy for daily off-site backup with the technique mentioned in 16.2, “Creating
a consistent point-in-time copy” on page 193, you can estimate the needed window to be the
sum of the application quiesce time, plus the FlashCopy establishment time, plus the
application restart time. However, the FlashCopy background copy operation may have an
influence over the Global Copy operation. In addition, if you take a backup to tape of the
FlashCopy target volumes, this activity also influences the Global Copy activity and the
FlashCopy background copy performance. Therefore, for a safer estimation of your daily
off-site backup window, as a very conservative approach we recommend that you estimate
each time sequentially, such as Global Copy data transmission, application quiesce,
FlashCopy establishment, application restart, FlashCopy background copy operation, and
backing up FlashCopy target.
You can estimate the approximate amount of data to be sent to the target by calculating the
amount of write data to the source. Tools to assist with this are operating system dependent,
like iostat. See the IBM Redbook IThe IBM TotalStorage DS8000 Series: Implementation,
SG24-6786. Another method, but not so exact, is to monitor the traffic over the FC switches
using FC switch tools and other management tools, but remember that only writes are
mirrored by Global Copy. You can also get some feeling about the proportion of reads/writes
by issuing datapath query devstats on SDD-attached servers.
Finally, you can estimate how much network bandwidth is necessary for your Global Copy
solution by dividing the amount of data you need to transfer by the duration of the backup
window.
The FlashCopy relationship at the remote site may influence the performance of the Global
Copy operation. If you use the FlashCopy with the nocopy option at the recovery site, when

the Global Copy target receives an update, the track on the FlashCopy source, which is also
the Global Copy target, has to be copied to the FlashCopy target before the data transfer
operation completes. This copy operation to the FlashCopy target can complete by using the
DS8000 cache and NVS without waiting for a physical write to the FlashCopy target.
However, this data movement may influence the Global Copy activity. So, when considering
the network bandwidth, consider that the FlashCopy effect over the Global Copy activity may
in fact decrease the bandwidth utilization during some intervals.
If not using the nocopy option, but resuming Global Copy before the FlashCopy background
copy finishes at the remote site, this background data movement may also influence the
Global Copy performance.
16.6 LSS design

Since the DS8000 has made the LSS a topological construct, which is not tied to a physical
Array as in the ESS, the design of your LSS layout can be simplified. It is now possible to
assign LSSs to applications, for example, without concern regarding under-allocation or
over-allocation of physical disk subsystem resources.
This may also help you when you use the freezepprc command. A freeze operation is
performed at the LSS level, causing all Global Copy volumes in that LSS to go into
suspended state with queue full condition and terminating all associated paths. If you assign
LSSs to each of your applications, you can control the impact of the queue full condition
caused by the freeze operation at an application level. If you put volumes used by several
applications into the same LSS, all those applications sharing the LSS, their Global Copy
volumes, will be affected by the queue full condition. You can issue one freezepprc
command to multiple LSSs. Therefore, you do not need to consolidate the number of LSSs
that your applications use.
In general, the freezepprc and unfreezepprc commands will be used in the Metro Mirror
environment or after converting from Global Copy mode to Metro Mirror mode. However, if
you set the Consistency Group option on an LSS, even when the volumes in the LSS are
Global Copy sources, the freeze operation triggered by the freezepprc command, or a failure
of a target update, causes the queue full condition. Therefore, you should pay attention to the
LSS design if you use the Consistency Group option.
16.7 Distance
The non-synchronous characteristics of Global Copy, combined with the great throughput
characteristic of its efficient track mirroring technique, make Global Copy well suited for
remote copy solutions at distances beyond the 300 km available with Metro Mirror. Global
Copy achieves greater distances without having to incur in the distance latency penalties that
synchronous write I/O methods exhibit.
The maximum distance for a direct Fibre Channel connection is 10 km. If you want to use
Global Copy over longer distances, the following connectivity technologies can be used to
extend this distance:
򐂰 Fibre Channel switches
򐂰 Channel Extenders over Wide Area Network (WAN) lines
򐂰 Dense Wave Division Multiplexors (DWDM) on dark fibre
Channel extender
Channel extender vendors connect DS8000 systems with a variety of Wide Area Network
(WAN) connections, including Fibre Channel, Ethernet/IP, ATM-OC3, and T1/T3.
When you use channel extender products with Global Copy, the channel extender vendor
determines the maximum distance supported between the source and target DS8000. The
channel extender vendor should be contacted for its distance capability, line quality
requirements, and WAN attachment capabilities.
A complete and current list of Global Copy supported environments, configurations, networks,
and products is available in the DS8000 Interoperability Matrix at:
You should also consult the channel extender vendor about hardware and software
prerequisites if you are using their products in a DS8000 Global Copy configuration.
Evaluation, qualification, approval, and support of Global Copy configurations using channel
extender products is the sole responsibility of the channel extender vendor.
Dense Wave Division Multiplexor (DWDM)

Wave Division Multiplexing (WDM) and Dense Wave Division Multiplexing (DWDM) comprise
the basic technology of fibre optical networking. The technique is used to carry many
separate and independent optical channels on a single dark fibre.
A simple way to envision DWDM is to consider that, at the sending end, multiple fibre optic
input channels (such as ESCON, Fibre Channel, FICON, or Gbit Ethernet) are combined by
the DWDM into a single fibre optic cable. Each channel is encoded as light for a different
wavelength. Think of each individual channel as an individual color — the DWDM system is
transmitting a rainbow. At the receiving end, the DWDM fans out the different optical
channels. DWDM, by the very nature of its operation, provides the full bandwidth capability of
the individual channel. As the wavelength of light is, from a practical perspective. infinitely
divisible, DWDM technology is only limited by the sensitivity of its receptors, as the total
aggregate bandwidth possible.
A complete and current list of Global Copy supported environments, configurations, networks,
and products is available in the DS8000 Interoperability Matrix at:
The DWDM vendor should be contacted regarding hardware and software prerequisites
when using their products in an DS8000 Global Copy configuration.
16.8 DS8000 configuration at the remote site

Depending on your operational environment, when using Global Copy you may also need
FlashCopy target volumes. If you need FlashCopy target volumes, you may be able to choose
higher capacity DDMs for the DS8000 at the remote site than those at the local site. However,
you should pay attention to the FlashCopy background copy performance mentioned in 16.5,
“Bandwidth” on page 197.
If you take backups of the FlashCopy target volumes onto tapes, you must ensure that the
tape resources are capable of handling these dump operations in between the point-in-time
checkpoints.

17
Chapter 17. Global Copy interfaces

The setup of Global Copy can be done using different interfaces. This chapter discusses the
interfaces that can be used for Global Copy management when Global Copy is used with the
IBM System Storage DS8000 in open systems environments.
The information discussed in the present chapter can be complemented with the following:
򐂰 Chapter 3, “DS Storage Manager (GUI)” on page 17
򐂰 Chapter 4, “DS Command-Line Interface” on page 23

17.1 Global Copy interfaces - overview
There are various interfaces available for the configuration and management of Global Copy
for DS8000 when used in an open systems environment:
This chapter gives an overview of the DS CLI and the DS SM for Global Copy management.
Also the following interfaces, not covered in this book, can be used:
򐂰 z/OS interfaces: TSO, ICKDSF, and ANTRQST API
In order to use a z/OS interface to manage open systems LUNs, the DS8000 must have at
least one CKD volume. If you are interested in this possibility, you can refer to the IBM
Redbook The IBM TotalStorage DS8000 Series: Copy Services with IBM Eserver zSeries,
SG24-6787.
For information about the DS Open API, refer to the publication IBM System Storage DS
Open Application Programming Interface Reference, GC35-0516.
DS CLI and DS SM similar functions for Global Copy management

Table 17-1 compares similar Global Copy commands from the DS CLI and the DS SM, and
the corresponding action.
Table 17-1 DS CLI and DS Storage Manager commands and actions

Task Command with Select option
DS CLI with DS SM
Global Copy paths commands
List available I/O ports that can be lsavailpprcport This information is shown
used to establish Global Copy during the process when a
paths. path is established.
List established Global Copy lspprcpath Copy Services → Paths

paths.
Establish path. mkpprcpath Copy Services → Paths →

Create
Delete path. rmpprcpath Copy Services → Paths →

Delete
Metro Mirror and Global Copy pairs commands
Failback. failbackpprc Copy Services → Global

Copy → Recover Failback
Failover. failoverpprc Copy Services → Global

Copy → Recover Failover
List Global Copy volume pairs. lspprc Copy Services → Global

Copy
Task Command with Select option
DS CLI with DS SM
Establish pair. mkpprc Copy Service → Global

Copy → Create
Suspend pair. pausepprc Copy Services → Global

Copy → Suspend
Resume pair. resumepprc Copy Services → Global

Copy → Resume
Delete pair. rmpprc Copy Services → Global

Copy → Delete
Freeze Consistency Group. freezepprc
Thaw Consistency Group. unfreezepprc
The DS CLI has the advantage that you can make scripts and use them for automation. The
DS Storage Manager GUI is a Web-based graphical user interface and is more intuitive than
the DS CLI.
17.2 Copy Services network components

To implement the Global Copy environment, we need the same network environment as for
Metro Mirror. You can refer to 14.2, “Copy Services network components” on page 135, for a
description.
17.3 DS Command-Line Interface (DS CLI)

You can use the DS CLI interface to manage all DS8000 Copy Services functions, such as
defining paths, establishing Global Copy pairs, and so on. For a detailed explanation about
the DS CLI, refer to Chapter 4, “DS Command-Line Interface” on page 23.
When you establish or remove Global Copy paths and volume pairs, you must give the DS
CLI commands to the DS HMC that is connected to the source DS8000. Also, when checking
status information at the local and remote site, you must issue DS CLI list type commands,
such as the lspprc, to each DS8000, source, and target.
In this section we describe and give examples of the available DS CLI commands that you
can use for Global Copy setup and control.
17.3.1 Global Copy path commands

You can use the following commands to establish and manage Global Copy paths.
Chapter 17. Global Copy interfaces 203

lsavailpprcport
The lsavailpprcport command displays the Fibre Channel ports that can be used to
establish Global Copy paths; see Example 17-1.
Example 17-1 lsavailpprcport command

dscli> lsavailpprcport -l -dev ibm.2107-75abtv1 -remotedev ibm.2107-7520781 -remotewwnn
5005076303FFC1A5 00:80
Date/Time: November 15, 2005 5:40:09 PM EET IBM DSCLI Version: 5.1.0.204 DS:
IBM.2107-75ABTV1
===================================================
lspprcpath
The lspprcpath command shows you the established paths from an LSS. In Example 17-2
we list the paths from LSS 00.
Example 17-2 lspprcpath command

dscli> lspprcpath -dev ibm.2107-75abtv1 00
IBM.2107-75ABTV1
=========================================================
00 80 Success 8000 I0010 I0143 5005076303FFC1A5
00 80 Success 8000 I0031 I0102 5005076303FFC1A5
mkpprcpath
With the mkpprcpath command, you can establish paths between two LSSs. In Example 17-3
we establish a path from 2107-75ABTV1 LSS 00 to 2107-7520781 LSS 80.
Example 17-3 mkpprcpath command

dscli> mkpprcpath -dev ibm.2107-75abtv1 -remotedev ibm.2107-7520781 -remotewwnn
5005076303FFC1A5 -srclss 00 -tgtlss 80 I0010:I0143 I0031:I0102
IBM.2107-75ABTV1
rmpprcpath
The rmpprcpath command removes all paths between two LSSs; see Example 17-4.
Example 17-4 rmpprcpath command

dscli> rmpprcpath -dev ibm.2107-75abtv1 -remotedev ibm.2107-7520781 00:80
Date/Time: November 15, 2005 5:51:11 PM EET IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-75ABTV1
CMUC00152W rmpprcpath: Are you sure you want to remove the Remote Mirror and Copy path 00:80:?
[y/n]:y
You can suppress confirmation message CMUC00152W by adding parameter -quiet in the
command.
17.3.2 Global Copy pair commands

You can use the following commands to establish and manage Global Copy pairs.
failbackpprc
You can issue the failbackpprc command against any Global Copy source volume that is in a
suspended state. The command copies the required data from the source volume to the
target volume in order to resume mirroring; see Example 17-5.
This command is usually used after a failoverpprc command that has been issued to restart
mirroring either in the reverse direction (recovery site to production site) or original direction
(production site to recovery site). The command in Example 17-5 is given to a former target
volume 808A that was changed to (source) suspended by a preceding failoverpprc command.
After the command completes, 808A becomes the source volume and 000A the target
volume. Note that because the command reverses the copy direction, you first have to
establish paths in that direction.
Example 17-5 failbackpprc command

dscli> failbackpprc -dev ibm.2107-7520781 -remotedev ibm.2107-75abtv1 -type gcp 808a:000a
Date/Time: November 16, 2005 10:23:04 AM EET IBM DSCLI Version: 5.1.0.204 DS:
IBM.2107-7520781
CMUC00197I failbackpprc: Remote Mirror and Copy pair 808A:000A successfully failed back.
failoverpprc
You can use the failoverpprc command to change a target device into (source) suspended
state while leaving the original source device in its current state; see Example 17-6.
Example 17-6 failoverpprc command

dscli> failoverpprc -dev ibm.2107-7520781 -remotedev ibm.2107-75abtv1 -type gcp 808a:000a
IBM.2107-7520781
CMUC00196I failoverpprc: Remote Mirror and Copy pair 808A:000A successfully reversed.
lspprc
The lspprc command shows you which volumes are in a Global Copy relationship; see
Example 17-7.
Example 17-7 lspprc command

dscli> lspprc -remotedev ibm.2107-7520781 -l 000a-000b
IBM.2107-75ABTV1
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade
===========================================================================================
000A:808A Suspended Host Source Global Copy 0 Disabled Disabled
000B:808B Copy Pending - Global Copy 0 Disabled Disabled

mkpprc
With the mkpprc command you can create a Global Copy pair. To create a Global Copy pair,
specify parameter -type gcp. See Example 17-8.
Example 17-8 mkpprc command

dscli> mkpprc -dev ibm.2107-75abtv1 -remotedev ibm.2107-7520781 -type gcp -mode full 000a:808a
Date/Time: November 16, 2005 11:02:55 AM EET IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-75ABTV1
CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 000A:808A successfully
created.
freezepprc
With the freezpprc command (see Example 17-9), you can stop the write activity on all
Global Copy source and target volumes for a given source and target LSS. The command
creates a new Consistency Group. It places the source logical subsystem (LSS) in the long
busy state so that no I/Os can be directed to it. It sets the source volumes on the source LSS
in suspended state. Target volumes are left in their current state. The command also removes
paths between the source LSS and target LSS and sets the queue full condition for the
source volume. This causes the host to queue writes to the source volume until the queue full
condition is reset (with the unfreezepprc command). During the queue full condition, the
source volume reports long busy status.
The command in Example 17-9 freezes all I/O to volume pairs whose source volume is on
LSS 00 of IBM.2107-75ABTV1 and the target on LSS 80 of IBM.2107-7520781.
Example 17-9 freezepprc command

dscli> freezepprc -dev ibm.2107-75abtv1 -remotedev ibm.2107-7520781 00:80
pausepprc
The pausepprc command suspends a Global Copy pair; see Example 17-10.
Example 17-10 pausepprc command
dscli> pausepprc -dev ibm.2107-75abtv1 -remotedev ibm.2107-7520781 000a:808a
CMUC00157I pausepprc: Remote Mirror and Copy volume pair 000A:808A relationship successfully
paused.
resumepprc
With the resumepprc command you can bring a pair from suspended to copy pending mode,
as shown in Example 17-11.
Example 17-11 resumepprc command

dscli> resumepprc -dev ibm.2107-75abtv1 -remotedev ibm.2107-7520781 -type gcp 000a:808a
CMUC00158I resumepprc: Remote Mirror and Copy volume pair 000A:808A relationship
successfully resumed. This message is being returned before the copy completes.
rmpprc
The rmpprc command removes the Global Copy relationship from a volume pair and sets the
volumes in simplex status; see Example 17-12.
Example 17-12 rmpprc command

dscli> rmpprc -dev ibm.2107-75abtv1 -remotedev ibm.2107-7520781 000a:808a
CMUC00160W rmpprc: Are you sure you want to delete the Remote Mirror and Copy volume pair
relationship 000A:808A:? [y/n]:y
CMUC00155I rmpprc: Remote Mirror and Copy volume pair 000A:808A relationship successfully
withdrawn.
You can suppress confirmation message CMUC00160W by adding parameter -quiet in the
command.
unfreezepprc
With the unfreezepprc command you can thaw an existing Consistency Group that was
created with the freeze command; see Example 17-13. It resumes host application I/O
activity for source volumes on the Consistency Group that was created by an earlier
freezepprc command. The unfreezepprc command is used to reset the queue full condition
for the source volume and release application I/O to the source volumes. All queued writes to
the source volume are written. The status of the volumes does not change; the source
volumes remain in the suspended state as set by the freezepprc command.
Example 17-13 unfreezepprc command
dscli> unfreezepprc -dev ibm.2107-75abtv1 -remotedev ibm.2107-7520781 00:80
Note: Because the freezepprc command deletes the paths between the specified LSS
pair, you have to re-establish them to resume normal Global Copy operation.
17.4 DS Storage Manager

You can use the DS Storage Manager (DS SM) Graphical User Interface (GUI) to set up and
control Global Copy. It is user friendly. However, you cannot use it for automation activities,
and certain Global Copy functions are not supported from this interface.
For a detailed explanation of the DS SM Graphical User Interface, refer to Chapter 3, “DS
Storage Manager (GUI)” on page 17.
Note: The DS SM GUI supports, differently from the DS CLI, a multi-session environment,
but not all functions and options are supported by the GUI.
We show examples using the DS Storage Manager in the following sections.

17.4.1 Path panel
The Path panel is the central point in the DS Storage Manager to establish new paths and
manage existing paths. You can get to the path panel as follows:
2. Select Copy Services.
3. Click Paths.
Then the Paths panel will display; see Figure 17-1. Here, to display existing paths, select the
storage complex, the storage unit, the storage image, and the source LSS for which you want
to display the paths information.
Figure 17-1 Paths panel
To create a new path, select Create from the Select Actions pull-down menu, and a process
will then guide you to establish a path.
Alternatively, if in the Paths panel you select one or more of the existing paths, then the Select
Actions pull-down menu will give you the following additional possible actions:
򐂰 Delete, to delete the paths you have selected before
򐂰 LSS copy options, to modify the copy options for an LSS (Figure 17-2)
Figure 17-2 LSS Copy Options panel
17.4.2 Metro Mirror panel
The Metro Mirror panel is the central point in the DS Storage Manager to establish new and
manage existing Global Copy pairs. You can get to the Metro Mirror panel as follows:
Then the Metro Mirror panel will display; see Figure 17-3.
Figure 17-3 Metro Mirror panel
To create a new Global Copy pair, choose Create from the Select Action pull-down menu.
The DS Storage Manager will guide you through the process to create a new pair.
If you select an existing Global Copy pair (see Figure 17-4), then the Select Action pull-down
menu will show you additional actions that you can perform with the selected volume pair:
򐂰 Delete: You can use this action to delete a Global Copy pair.
򐂰 Suspend: You can use this action to suspend a Global Copy pair.

򐂰 Convert to synchronous: You can use this action to convert a Global Copy pair into a
synchronous Metro Mirror pair.
򐂰 Resume: You can use this action to resume a suspended Global Copy pair.
򐂰 Recover Failover: You can use this action to change a target device into a (source)
suspended device while leaving the original source device in its current state.
򐂰 Recover Failback: You can use this action to switch back to the original site after a failover.
򐂰 Properties: You can use this action to view the volume pair properties, including the
number of out-of-sync tracks.
18
Chapter 18. Global Copy performance and

scalability
In this chapter we discuss performance and scalability considerations when using Global
Copy with the DS8000.

18.1 Performance
As the distance between DS8000s increases, Metro Mirror response time is proportionally
affected—and this negatively impacts the application performance. When implementations
over extended distances are needed, Global Copy becomes an excellent trade-off solution.
You can estimate Global Copy application impact as that of the application when working with
Metro Mirror suspended volumes. For the DS8000, there is some more work to do with the
Global Copy volumes, as compared to the suspended volumes, because with Global Copy,
the changes have to be sent to the remote DS8000. But this is a negligible overhead for the
application, as compared with the typical synchronous overhead.
There will be no processor resources (CPU and memory) consumed by the Global Copy
volume pairs, as this is managed by the DS8000 subsystem.
If you take FlashCopy at the recovery site in your Global Copy implementation, you should
take into account the influence between Global Copy and the FlashCopy background copy;
refer to 16.5, “Bandwidth” on page 197.
18.2 Scalability
The DS8000 Global Copy environment can be scaled up or down as required. If new volumes
are added to the DS8000 that require mirroring, they can be dynamically added. If additional
Global Copy paths are required, they also can be dynamically added.
18.2.1 Adding capacity

As we have previously mentioned, the logical nature of the LSS has made a Global Copy
implementation on the DS8000 easier to plan, implement, and manage. However, if you need
to add more LSSs to your Global Copy environment, your management and automation
solutions should be set up to handle this.
The eRCMF service offerings are designed to provide this functionality. Visit the IBM web site
and see the Services and Industry Solutions page for more information.
Adding capacity to same DS8000

If you are adding capacity to an existing DS8000, provided that your Global Copy link
bandwidth is not close to or over its limit, you may only need to add volume pairs into your
configuration. If you are adding more LSSs, then you will need to define Global Copy paths
before adding volume pairs. Keep in mind that when you add capacity for Global Copy use,
you might have to acquire a new license feature code for Global Copy that corresponds to the
new capacity.
Adding capacity in new DS8000s

If you are adding new DS8000s into your configuration, you will need to add physical links
prior to defining your Global Copy paths and volume pairs. We recommend a minimum of two
paths per DS8000 pair for redundancy reasons. Your bandwidth analysis will indicate if you
require more than two paths.
19
Chapter 19. Global Copy examples

In this chapter we provide examples of how to configure and manage Global Copy on the
DS8000 by using the DS CLI and the DS GUI. This includes:
򐂰 Set up a Global Copy environment.
򐂰 Remove a Global Copy environment.
򐂰 Maintain the Global Copy environment.
򐂰 Periodical off-site backup procedure.
򐂰 Use the DS GUI to manage Global Copy.

19.1 Set up Global Copy environment using DS-CLI
In the following sections we present an example of how to se up a Global Copy environment
using the DS CLI. Figure 19-1 shows the configuration that we implement.
LSS10 LSS20
1000 2000
1001 2001
LSS11 Physical Fibre path

LSS21
1100 2100
1101 2101
DS8000#1 DS8000#2
Figure 19-1 DS8000 configuration in the Global Copy set up example
In our example we use different LSS and LUN numbers for the Global Copy source and target
elements, so that you can more clearly understand which one is being specified when going
through the reading of the example.
Note: In a real environment, and differently from our example, to simplify the management
of your Global Copy environment, we recommend that you maintain a symmetrical
configuration in terms of both physical and logical elements.
The procedure to set up the Global Copy environment is similar to that of a Metro Mirror
environment. One thing that is different is the type of relationship that is established between
the volumes. In this example we define Global Copy pairs.

As we prepare to work with the DS CLI we do some initial tasks that will simplify our activities
during the configuration process. Refer to 14.5.1, “Preparing to work with the DS CLI” on
page 137.
19.1.2 Set up of Global Copy configuration
Figure 19-2 shows the configuration we set up for this example. The configuration has four
Global Copy pairs that reside in two LSSs. Two paths are defined between each source and
target LSS.
Source Paths Target

LSS10 LSS20
1000 2000
Global Copy pairs
1001 2001
LSS11 FCP links

LSS21
1100 2100
Global Copy pairs
1101 2101
DS8000#1 DS8000#2
Figure 19-2 Global Copy environment to set up
To configure the Global Copy environment, we follow this procedure:

1. Determine the available Fibre Channel links for paths definition.
2. Define the paths that Global Copy will use.
3. Create Global Copy pairs.
19.1.3 Determine the available Fibre Channel links

This step is similar to the Metro Mirror setup example. Refer to 14.5.3, “Determine the
available Fibre Channel links” on page 139.
19.1.4 Create paths for Global Copy

This step is similar to for the Metro Mirror example. Refer to 14.5.4, “Create Metro Mirror
paths” on page 139.
19.1.5 Create Global Copy pairs

After creating the paths, you can establish the Global Copy volume pairs. This is done with
the mkpprc command and verifying the result with the lspprc command; see Example 19-1.
When you create a Global Copy pair, you specify the -type gcp parameter with mkpprc.
Example 19-1 Create Global Copy pairs and verify the result
dscli> mkpprc -remotedev IBM.2107-75ABTV1 -type gcp 1000-1001:2000-2001 1100-1101:2100-2101
Date/Time: November 2, 2005 10:53:22 PM JST IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-7520781
dscli>
dscli> lspprc 1000-1001 1100-1101
Chapter 19. Global Copy examples 215

==================================================================================================
1000:2000 Copy Pending - Global Copy 10 unknown Disabled False
You can check the status of Global Copy using the lspprc -l command. Out Of Sync Tracks
shows the remaining tracks to be sent to the target volume (the size of the logical track for the
FB volume for the DS8000 is 64 KB). You can use the lspprc -fullid command flag to
display the fully qualified DS8000 storage image ID in the command output; see
Example 19-2.
Example 19-2 lspprc -l and lspprc -fullid for Global Copy pairs
dscli> lspprc -l 1000-1001 1100-1101
==========================================================================================================================================
1000:2000 Copy Pending - Global Copy 38379 Disabled Disabled invalid - 10 unknown
Disabled False
Disabled False
Disabled False
Disabled False
dscli>
dscli> lspprc -fullid 1000-1001 1100-1101
Status
==========================================================================================================================================
====
IBM.2107-7520781/1000:IBM.2107-75ABTV1/2000 Copy Pending - Global Copy IBM.2107-7520781/10 unknown Disabled True
Unlike the Metro Mirror initial copy (first pass), the state of the Global Copy volumes still
shows Copy Pending even after the Out Of Sync Tracks become 0; see Example 19-3.
Example 19-3 List the Global Copy pairs status after Global Copy first pass completes
dscli> lspprc -l 1000-1001 1100-1101
==========================================================================================================================================

Disabled True
Disabled True
Disabled True
Disabled True
dscli>
Copy Pending is the state of the Global Copy source. The target state is Target Copy
Pending. To see the target state in this example, you have to give the lspprc command to the
HMC connected to DS8000#2, which is the Global Copy target. See Example 19-4.
Example 19-4 lspprc for Global Copy target volumes

dscli> lspprc 2000-2001 2100-2101
Date/Time: November 2, 2005 10:53:47 PM JST IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-75ABTV1
=========================================================================================================
1000:2000 Target Copy Pending - Global Copy 10 unknown Disabled Invalid
19.2 Remove Global Copy environment using DS CLI

This step is similar to the Metro Mirror example (see 14.6, “Remove Metro Mirror environment
using DS CLI” on page 141). However, here we show examples for Global Copy because the
outputs of the lspprc command are different from that in the Metro Mirror environment.
In general, the following steps should be done to clear the Global Copy environment.
1. Remove Global Copy pairs.
2. Remove logical paths.
19.2.1 Remove Global Copy pairs

The rmpprc command removes a volume pair relationship; see Example 19-5. You can use
the -quiet parameter to turn off the confirmation prompt for this command.
Example 19-5 Removing Global Copy pairs

dscli> lspprc 1000-1001 1100-1101
==================================================================================================
1000:2000 Copy Pending - Global Copy 10 unknown Disabled True
dscli>
dscli> rmpprc -remotedev IBM.2107-75ABTV1 1000-1001:2000-2001 1100-1101:2100-2101
1000-1001:2000-2001:? [y/n]:y
1100-1101:2100-2101:? [y/n]:y

You can add the -at tgt parameter to the rmpprc command to remove only the available
Global Copy target volumes; see Example 19-6. In Example 19-6, you have to issue this
command to the HMC connected to DS8000#2, which is the Global Copy target.
Example 19-6 Results of rmpprc with -at tgt

dscli> lspprc 2002
=========================================================================================================
dscli>
dscli> rmpprc -remotedev IBM.2107-75ABTV1 -quiet -at tgt 1002:2002
dscli>
dscli> lspprc 2002
Example 19-7 shows the Global Copy source volume status after the rmpprc -at tgt
command has completed and it also shows the result of a rmpprc -at src command. In this
case, there were still available paths. Therefore, the source volume state changed after the
rmpprc -at tgt command completed. If there were no available paths, the state of the Global
Copy source volumes would have been preserved. In Example 19-7, you must issue this
command to the HMC connected to DS8000#1, which is the Global Copy source.
Example 19-7 Global Copy source volume status after rmpprc with -at tgt and rmpprc with -at src
dscli> lspprc 1002
==================================================================================================
<< After rmpprc -at tgt command completes >>
dscli> lspprc 1002

ID State Reason Type SourceLSS Timeout (secs) Critical Mode First Pass Stat
=====================================================================================================
1002:2002 Suspended Simplex Target Global Copy 10 unknown Disabled True
dscli>
dscli> rmpprc -remotedev IBM.2107-75ABTV1 -quiet -at src 1002:2002
dscli>
dscli> lspprc 1002
19.2.2 Remove paths

The rmpprcpath command removes the paths. Before removing the paths, you must remove
all volume pairs that are using the paths or you have to use the -force parameter with the
rmpprcpath command; see Example 19-8.
Example 19-8 Remove paths

dscli> lspprc 1000-1001 1100-1101
Date/Time: November 3, 2005 1:27:34 AM JST IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-7520781
==================================================================================================
dscli>
dscli> rmpprc -remotedev IBM.2107-75ABTV1 -quiet 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
dscli> lspprc 1000-1001 1100-1101
dscli>
dscli> rmpprcpath -quiet -remotedev IBM.2107-75ABTV1 10:20 11:21
If you do not remove the Global Copy pairs that are using the paths, the rmpprcpath
command fails; see Example 19-9.
Example 19-9 Remove paths without removing the Global Copy pairs
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
If you want to remove the paths still having the Global Copy pairs, you can use the -force
parameter; see Example 19-10. After the path has been removed, the Global Copy pair is still
in copy pending state until the source receives I/O from the servers. When I/O goes to the
Global Copy source, the source volume becomes suspended. If you set the Consistency
Group option for the LSS in which the volumes reside, I/Os from the servers are held with
queue full status for the specified timeout value.
Example 19-10 Remove paths still having Global Copy pairs - use -force parameter
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli> rmpprcpath -quiet -remotedev IBM.2107-75ABTV1 -force 10:20 11:21

dscli>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
<< After I/O goes to the source volume(1000 and 1001) >>
dscli> lspprc 1000-1001 1100-1101

Status
=====================================================================================================================
=
1000:2000 Suspended Internal Conditions Target Global Copy 10 unknown Disabled True
1001:2001 Suspended Internal Conditions Target Global Copy 10 unknown Disabled True
19.3 Maintain Global Copy environment using DS CLI

This section shows how we can manage the Global Copy environment, such as suspend and
resume Global Copy pairs, change the mode from Global Copy to Metro Mirror, and change
paths.
19.3.1 Suspend and resume Global Copy data transfer

The pausepprc command stops transferring data to the Global Copy target volume. After this
command completes, the Global Copy pair becomes suspended. See Example 19-11.
Example 19-11 Suspend Global Copy data transfer

dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
dscli> pausepprc -remotedev IBM.2107-75ABTV1 1000-1001:2000-2001
dscli>
dscli> lspprc 1000-1001 1100-1101
=======================================================================================================
1000:2000 Suspended Host Source Global Copy 10 unknown Disabled True
Because the source DS8000 keeps records of all changed data on the source volume, you
can resume Global Copy data transfer at a later time. The resumepprc command resumes a
Global Copy relationship for a volume pair and restarts transferring data. You must specify the
copy mode such as Metro Mirror or Global Copy with the -type parameter; see
Example 19-12.
Example 19-12 Resume Global Copy pairs

dscli> lspprc 1000-1001 1100-1101
=======================================================================================================
dscli>
dscli> resumepprc -remotedev IBM.2107-75ABTV1 -type gcp 1000-1001:2000-2001
dscli>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
19.3.2 Change copy mode from Global Copy to Metro Mirror

You can change the copy mode from Global Copy to Metro Mirror with the mkpprc command;
see Example 19-13. This operation is called Go-to-sync. Depending on the amount of data to
be sent to the target, it takes time until the pairs become full duplex.
Example 19-13 Change copy mode from Global Copy to Metro Mirror
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
dscli> mkpprc -remotedev IBM.2107-75ABTV1 -type mmir 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================

You can use the -suspend parameter with the mkpprc -type mmir command. If you use this
parameter, the state of the pairs becomes suspended when the data synchronization is
completed; see Example 19-14. You can use this option for your off-site backup scenario with
the Global Copy function.
Example 19-14 mkpprc -type mmir with -suspend

dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
dscli> mkpprc -remotedev IBM.2107-75ABTV1 -type mmir -suspend 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
dscli> lspprc 1000-1001 1100-1101
=====================================================================================================
You can add the -wait parameter with the mkpprc command. With the -wait parameter, the
mkpprc -type mmir -suspend command does not return to the command prompt until the
pairs complete data synchronization and reach the suspended state; see Example 19-15.
Note: If you do not specify the -wait parameter with the mkpprc -type mmir -suspend com-
mand, the mkpprc command does not wait for the data synchronization. If you do not use
the -wait parameter, you must check the completion of the synchronization with the lspprc
command.
Example 19-15 mkpprc -type mmir -suspend with -wait

dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
dscli> mkpprc -remotedev IBM.2107-75ABTV1 -type mmir -suspend -wait 1000-1001:2000-2001
1100-1101:2100-2101
1/4 pair 1001:2001 state: Suspended
dscli>
dscli> lspprc 1000-1001 1100-1101
=====================================================================================================
19.3.3 Change the copy mode from Metro Mirror to Global Copy
You cannot change the copy mode from the Metro Mirror to Global Copy directly. In order to
do this, you must use the pausepprc command to suspend the Metro Mirror pair first, and then
resume the pair in Global Copy mode with the resumepprc -type gcp command; see
Example 19-16.
Example 19-16 Change copy mode from Metro Mirror to Global Copy
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
CMUN03053E mkpprc: 1000:2000: Copy Services operation failure: invalid transition
dscli>
dscli> pausepprc -remotedev IBM.2107-75ABTV1 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
dscli> lspprc 1000-1001 1100-1101
=====================================================================================================
dscli>
dscli> resumepprc -remotedev IBM.2107-75ABTV1 -type gcp 1000-1001:2000-2001 1100-1101:2100-2101

dscli>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
19.3.4 Add and remove paths

This step is similar to the Metro Mirror example. Refer to 14.7.2, “Add and remove paths” on
page 145.
19.4 Periodic off-site backup procedure

This section shows how to control the procedure discussed in 16.2, “Creating a consistent
point-in-time copy” on page 193, to use Global Copy for periodical off-site backup.
Figure 19-3 shows a diagram of the DS8000 Global Copy environment for this example. We
use the Remote Incremental FlashCopy function for the FlashCopy at the recovery site
(DS8000#2). We use 1000, 1001, 1100, and 1101 in DS8000#1 for Global Copy (GC)
sources; 2000, 2001, 2100, and 2101 in DS8000#2 for Global Copy targets and FlashCopy
(FC) sources; and 2002, 2003, 2102, and 2103 in the DS8000#2 for FlashCopy targets.
GC Target
and
GC Source FC source FC Target
Paths
LSS10 LSS20 FlashCopy
1000 2000 2002

Global Copy pairs
1001 2001 2003
LSS11 FCP links LSS21 FlashCopy
1100 2100 2102

Global Copy pairs
1101 2101 2103
DS8000#1 DS8000#2
Figure 19-3 The DS8000 environment for Global Copy offsite backup
19.4.1 Initial setup for this environment
In order to set up the Global Copy and FlashCopy environment, we follow these steps:
1. Create paths, create pairs, and wait for the completion of the Global Copy initial copy.
2. Create the Incremental FlashCopy relationship at the recovery site and wait for the
completion of the FlashCopy background copy.
Step 1 - create paths and pairs

An example of this step is shown in Example 19-17.
Example 19-17 Step 1 of the initial setup

i0143:i0010 i0213:i0140
i0143:i0010 i0213:i0140
dscli>
dscli>
dscli> lspprc -l 1000-1001 1100-1101
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended
SourceLSS Timeout (secs) Critical Mode First Pass Status
=====================================================================================================================
================================================
1000:2000 Copy Pending - Global Copy 0 Disabled Disabled invalid - 10

unknown Disabled True
Step 2 - create FlashCopy relationship

An example of this step is shown in Example 19-18.
Example 19-18 Step 2 of the initial setup

dscli> mkremoteflash -record -persist -conduit IBM.2107-7520781/10 -dev IBM.2107-75ABTV1
2000-2001:2002-2003
Date/Time: November 3, 2005 2:40:01 AM JST IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-75ABTV1
CMUC00173I mkremoteflash: Remote FlashCopy volume pair 2000:2002 successfully created. Use the
lsremoteflash command to determine copy completion.
dscli>
dscli> mkremoteflash -record -persist -conduit IBM.2107-7520781/11 -dev IBM.2107-75ABTV1
2100-2101:2102-2103

We then verify the FlashCopy background copy completion; see Example 19-19.
Example 19-19 lsremotefrash to check the FlashCopy background copy completion

dscli> lsremoteflash -l -conduit IBM.2107-7520781/10 -dev IBM.2107-75ABTV1 2000-2001
ID SrcLSS SequenceNum ActiveCopy Recording Persistent Revertible SourceWriteEnabled TargetWriteEnabled BackgroundCopy OutOfSyncTracks
==========================================================================================================================================
2000:2002 20 0 Enabled Enabled Enabled Disabled Enabled Enabled Enabled 44626
dscli>
==========================================================================================================================================
dscli>
=========================================================================================================================================
2000:2002 20 0 Disabled Enabled Enabled Disabled Enabled Enabled Enabled 0
dscli>
==========================================================================================================================================
19.4.2 Periodical backup operation
Depending on your backup requirement and application acceptance, you schedule the
following scenario periodically, such as daily. We show examples of the DS CLI commands to
control this procedure, based on the scenario illustrated in Figure 19-4.
Application running
1.Updated data is being sent

in normal Global Copy
source target
A volume B volume
C volume
Quiesce Application
2.Quiesce Application
source target
3.Go-to-sync and suspend

volume pairs
source target
Restart Application
4.Restart Application
source target Now we have
consistent
Application running data
5.Take FlashCopy B to C. FlashCopy
Return to 1.
source target
Production site Recovery site

Figure 19-4 Global Copy offsite backup scenario
The explanation of the steps is the following:

1. Normal Global Copy mode operation.
2. Quiesce the application at the production site.
3. Go-to-sync and suspend the Global Copy pairs.
4. Restart the application at the production site.
5. Take a FlashCopy B to C.
For a detailed description of the preceding steps refer to 16.2.1, “Procedure to take a
consistent point-in-time copy” on page 194. Next we see how you can execute the procedure.
Step 1
In this step we are operate in the normal Global Copy mode. The lspprc and lsremoteflash
commands show the status in Example 19-20.
Example 19-20 Step 1 of the periodical operation

dscli> lspprc 1000-1001 1100-1101
==================================================================================================

dscli> lsremoteflash -conduit IBM.2107-7520781/10 -dev IBM.2107-75ABTV1 2000-2001
ID SrcLSS SequenceNum ActiveCopy Recording Persistent Revertible SourceWriteEnabled TargetWriteEnabled BackgroundCopy
============================================================================================================================
2000:2002 20 0 Disabled Enabled Enabled Disabled Enabled Enabled Enabled
ID SrcLSS SequenceNum ActiveCopy Recording Persistent Revertible SourceWriteEnabled TargetWriteEnabled BackgroundCopy
============================================================================================================================
Step2
Depending on your application and platform, you should take the necessary actions to briefly
stop updates to the source volumes — quiesce application.
Step3
The DS CLI command examples for this step are shown in Example 19-21.
Example 19-21 The mkpprc -type mmir -wait -suspend

dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
dscli> mkpprc -remotedev IBM.2107-75ABTV1 -type mmir -suspend -wait 1000-1001:2000-2001 1100-1101:2100-2101
<< It may take time to show bellow messages depending on the amount of the Out Of Sync Tracks remained. >>

dscli>
dscli> lspprc 1000-1001 1100-1101
=====================================================================================================
dscli> lspprc -l 1000-1001 1100-1101
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS
========================================================================================================================
1000:2000 Suspended Host Source Metro Mirror 0 Disabled Disabled invalid - 10
dscli>
Step 4
Depending on your application and platform, you should take the necessary actions to restart
your application — resume I/O activity to the source volumes.
Step 5
The DS CLI command examples for this step are shown in Example 19-22 and
Example 19-23. We use the resyncremoteflash command to perform the Incremental
FlashCopy from the production site. We add the -seqnum parameter with the mkremoteflash
to add the FlashCopy sequence number to identify this FlashCopy set.
We specify the -type gcp parameter with the resumepprc command to restart the normal
Global Copy mode.
Example 19-22 Take FlashCopy B to C

dscli> resyncremoteflash -record -persist -seqnum 20051103 -conduit IBM.2107-7520781/10 -dev
IBM.2107-75ABTV1 2000-2001:2002-2003
CMUC00175I resyncremoteflash: Remote FlashCopy volume pair 2000:2002 successfully resynchronized. Use the
dscli> resyncremoteflash -record -persist -seqnum 20051103 -conduit IBM.2107-7520781/11 -dev
IBM.2107-75ABTV1 2100-2101:2102-2103
dscli>
ID SrcLSS SequenceNum ActiveCopy Recording Persistent Revertible SourceWriteEnabled TargetWriteEnabled Background
========================================================================================================================
dscli>
========================================================================================================================
Example 19-23 Resume normal Global Copy mode

dscli> lspprc 1000-1001 1100-1101
=====================================================================================================
dscli>
dscli> resumepprc -remotedev IBM.2107-75ABTV1 -type gcp 1000-1001:2000-2001 1100-1101:2100-2101

CMUC00158I resumepprc: Remote Mirror and Copy volume pair 1000:2000 relationship successfully resumed. This
message is being returned before the copy completes.
dscli>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================

In these examples we use the DS Storage Manager Graphical User Interface to establish
paths and then establish and manage the Global Copy pairs.
Note: To create and monitor Global Copy pairs, use the Metro Mirror panels.
19.5.1 Establish paths with the DS Storage Manager GUI

To establish a new path from one LSS to another LSS you follow this procedure:
3. Click Paths.
4. Select your source Storage Complex, Storage Unit, Storage Image, and LSS.
The Paths panel is displayed; see Figure 19-5. Because there are no listed paths, we know
that LSS 14 has no paths defined that use LSS 14 as the source LSS. In our example we
want to configure two paths from the DS8000 serial 7503461 LSS 14 to the DS8000 serial
7520781 LSS 18. From the Select Actions pull-down menu select Create.
Figure 19-5 Paths panel
We can now select the source LSS from the production DS8000; see Figure 19-6.
Figure 19-6 Select source LSS
When finished click Next.

The Select target LSS panel is displayed; see Figure 19-7. Here we select the target Storage
Complex, Storage Unit, Storage Image, and target LSS.
Figure 19-7 Select target LSS
If your intended storage unit is not presented as an alternative then you must add the storage
complex that contains that storage unit. See Chapter 3, “DS Storage Manager (GUI)” on
page 17. First add the storage complex and then restart the process to create paths.
Click Next when finished.
The Select source I/O ports panel is displayed. Here you can select the source I/O ports; see
Figure 19-8. Because we want to establish two paths, we select both I/O ports.
Figure 19-8 Selecting source ports
We then click Next.
The Select target I/O ports panel is displayed. Here you select the target ports. Because there
is only one target port available for each source port, the choices are made for us, as shown
in Figure 19-9.
Figure 19-9 Select target I/O ports
The Select path options panel is displayed; see Figure 19-10. Here you can decide if you
want to create a Consistency Group. Because this is the setup of a Global Copy environment
we do not need a Consistency Group; therefore, we do not select this option.
Figure 19-10 Select path options

The Verification panel is displayed; see Figure 19-11. Here we can verify the information we
entered and, if necessary, click Back to do corrections or click Finish to create the logical
paths.
Figure 19-11 Verification panel
After you finish you are again presented the Paths panel, and now you will see the existing
paths that you have just created; see Figure 19-12.
Figure 19-12 Paths defined
19.5.2 Establish Global Copy pairs

In this section we establish the Global Copy pairs. Once the volumes are in the copy pending
state, we synchronize the volumes to change the status to full duplex.
3. Select Metro Mirror.
The Metro Mirror panel will be displayed. See Figure 19-13. Here you select the source
Storage Complex, Storage Unit, Storage Image, and LSS. Then from the Select Action
pull-down menu select Create.
The Volume Pairing Method panel will be displayed; see Figure 19-14. Here we must select
either Automated volume pair assignment or Manual volume pair assignment. In our example
we select Automated volume pair assignment.
Figure 19-14 Volume Paring Method

The Select source volumes panel will be displayed; see Figure 19-15. Here we select the
source volumes 1400, 1401, and 1402.
Figure 19-15 Select source volumes
The Select target volumes (Auto pairing) panel will be displayed; see Figure 19-16. Here we
select the target Storage Complex, Storage Unit, Storage Image, and Resource Type. In our
example we chose the resource type LSS, so now we have to select the LSS number. After
this, we select the target volumes, which, because we chose automated pairing in the
previous step, will be automatically paired to source volumes.
Figure 19-16 Select target volumes
The Select copy options panel will be displayed; see Figure 19-17. Here we select Global
Copy and we also select Perform initial copy.
Figure 19-17 Copy options
The Verification panel will be displayed; see Figure 19-18. Here we can verify the volume
pairs we configured and, if necessary, click Back to do corrections or click Finish to create
the Global Copy pairs.
Figure 19-18 Verification

After you finish you are again presented the Metro Mirror panel, and now you will see the
existing Global Copy pairs that were just created. The State column for these volumes will
show copy pending. See Figure 19-19.
Figure 19-19 Global Copy volumes in copy pending status
19.5.3 Monitoring the copy status

You may want to see how many tracks of data must be copied from the source to the target
volume. To do this you must select the volume pair in the Metro Mirror panel, and then from
the Select Actions pull-down menu choose Properties. A Properties panel will be displayed;
see Figure 19-20. Here if you select the Out-of-sync tracks tab, you will see the number of
tracks that must still be copied. In our example you can see that there are zero tracks to copy.
Figure 19-20 Out of sync tracks
19.5.4 Convert to Metro Mirror (synchronous)
To have the volume pairs in a full duplex state, we need to synchronize them. In the Metro
Mirror panel we select the volumes we want to synchronize and then from the Select Actions
pull-down menu we choose Convert to synchronous. See Figure 19-21.
Figure 19-21 Selecting the action to convert to synchronous
The Convert to synchronous confirmation panel will be displayed; see Figure 19-22. This
panel shows all the select volumes. Note that the Type column shows Metro Mirror. This is
because we are converting from non-synchronous Global Copy to synchronous Metro Mirror.
Figure 19-22 Convert to synchronous confirmation panel
If everything is correct and you want to proceed with the synchronization, click OK to
continue.

Depending on the amount of write operations on the source and the available Fibre Channel
links bandwidth, the time for the volumes to reach the full duplex state will vary. In the Metro
Mirror panel, keep clicking Refresh until the State column changes to full duplex; see
Figure 19-23.
19.5.5 Suspend a pair

After the volumes are in duplex state we can stop the I/O for a short time on the production
site to ensure that we have data consistency at the remote site. After the I/O has stopped we
can suspend the volumes.
In the Metro Mirror panel we select the volumes we want to suspend and then we choose
Suspend from the Select Action pull-down. We then get the choice to suspend the volumes
at the source or at the target. Normally we suspend at the source. We then click OK and the
volumes will suspend. Figure 19-24 shows the volumes already in suspended state.
Once the volumes are suspended we can restart the I/O at the application site. We can also
make a FlashCopy of the target volumes at the recovery site, and when finished we could
resume the pairs to re-establish the Global Copy replication again.
Part 6
Part 6 Global Mirror

This part of the book describes the IBM System Storage Global Mirror when used in open
systems environments with the DS8000. We discuss the characteristics of Global Mirror and
describe the options for its setup. We also show which management interfaces can be used,
as well as the important aspects to be considered when establishing a Global Mirror
environment. This part ends with examples of Global Mirror management and setup.
The following are the topics covered in this part:

򐂰 Global Mirror overview
򐂰 Global Mirror options and configuration
򐂰 Global Mirror interfaces
򐂰 Performance and scalability
򐂰 Examples

20
Chapter 20. Global Mirror overview

In this chapter we provide an overview of what Global Mirror is. We also discuss the need for
data consistency at a distant site when synchronous data replication such as Metro Mirror is
not possible. This chapter then explains how Global Mirror works in a similar manner to a
distributed application, in a server and client relationship. Finally, this chapter contains a
step-by-step process to establish a Global Mirror environment.
The information discussed in this chapter is complemented with the following IBM
publications and IBM Redbooks:
򐂰 The IBM TotalStorage DS8000 Series: Implementation, SG24-6786
򐂰 IBM System Storage DS8000 Command-Line Interface User´s Guide, SC26-7916

20.1 Synchronous and non synchronous data replication
When replicating data over long distances, beyond 300 km, asynchronous data replication is
the valid approach. This is basically so because with the asynchronous techniques, the
application I/O processing at the local storage disk subsystem remains independent of the
process of data transmission to the remote storage disk subsystem.
Still with the asynchronous data replication techniques, we must provide additional means to
ensure data consistency at the remote location. It even requires a solution that guarantees
data consistency not only within a single local-remote pair of storage disk subsystems but
also across multiple local and remote storage disk subsystems.
For a given pair of local and remote storage disk subsystems, a time stamp approach leads to
consistent data at the remote storage disk subsystem. Using this approach, and by sorting
the I/Os by their time stamps, the write I/Os can be applied at the remote disk subsystem in
the same sequence as they arrived at the local disk subsystem.
But when the application volumes are spread across multiple storage disk subsystems, this
time stamp concept alone is not sufficient to replicate data and provide data consistency at
the remote site. This additionally requires a Consistency Group concept.
In the rest of the present section we discuss how consistent data, that is dependent writes,
are managed either with a synchronous technique like Metro Mirror versus different
asynchronous techniques like Global Copy or Global Mirror.
20.1.1 Synchronous data replication and dependent writes

In normal operations, the nature of synchronous data replication preserves data consistency
for dependent writes. Dependent writes and data consistency are explained in detail in 12.4,
“Consistency Group function” on page 116.
Host
Server
1 4
Storage Disk Storage Disk
Subsystem 1 Subsystem 2
A
2C00
A1
A
Primary
Primary
2 Synchronous B1
Primary
Primary Replicate
source 3 target
local remote
Figure 20-1 Synchronous data replication
In synchronous data replication methods such as Metro Mirror, an application write always
goes through the following four steps; see Figure 20-1:
1. Write the data to the source storage disk subsystem cache. Note that this does not end
the I/O and does not present an I/O complete to the application.
2. Replicate the data from the source storage disk subsystem cache to the target storage
disk subsystem cache.
3. Acknowledge to the source storage disk subsystem that data successfully arrived at the
target storage disk subsystem.
4. Present the successful I/O completion to the server, which is presented to the application
and concludes this I/O.
Now the next I/O that depends on the successful completion of the previous I/O can be
issued.
When you have dependent writes across multiple storage disk subsystems, synchronous
data replication alone does not guarantee that you can restart applications at the remote site
without doing some previous data recovery.
Consider a database environment that spreads across multiple storage disk subsystems at
the local or remote site; see Figure 20-2.
Database Database
4
Subsystem Subsystem
Switch over to remote site
source 1 target
2 Restart DB Subsystem 5

3 Recover A2
A
2C00 3C00
A1
A
Primary
Primary Synchronous B1
Primary
Log
Primary
1 Log' 6
Replicate
Recover A2
2D00
A
A
2D00
A
3D00
A
A2
Primary
Primary
Primary
Primary
Primary
Synchronous
B2
Primary
Primary
Primary Replicate Primary
DB DB'

2D00
A
2E00
Synchronous A
3E00
A
Primary
A3
Primary
Primary
Primary
Primary Replicate
3 B3
Primary
Primary
Primary Primary
RECON RECON'
Figure 20-2 Synchronous data replication and unnecessary recovery after local site failure
A database update usually involves three dependent write I/Os to ensure data consistency,
even in the event of a failure during this process: see Figure 20-2:
1. Update intent to the logging files. Logging may happen to two logging files.
2. Update the data.
3. Indicate update complete to the logging files.
This sequence of I/Os is also called a two-phase commit process.
Chapter 20. Global Mirror overview 245

When you are in a remote copy environment, in an outage situation, the following sequence
may occur; see Figure 20-2 on page 245:
1. Write update intent to logging volume A1.
2. Update to database on volume A2 eventually fails due to a replication problem from the
local site to the remote site. This may also be the beginning of a rolling disaster.
3. The database subsystem recognizes the failed write I/O and indicates, in a configuration
file on volume A3, that one or more databases on A2 have to be recovered due to an I/O
error. This gets replicated to the remote site because the paths for this storage disk
subsystem pair are still working and this particular source storage disk subsystem did not
fail yet.
4. The failure eventually progressed and failed the local site completely. The database
subsystem is restarted at the remote site after switching over to the remote site.
5. At startup, the database subsystem discovers in its configuration file that the database on
A2-B2 has to be recovered as indicated before. This is actually not necessary because the
data was synchronously replicated and both related volumes, A2 and B2, are identical and
at the very same level of data currency — that of the moment previous to the error.
Nevertheless, the database subsystem will still recover all databases that are marked for
recovery as per the information found in the configuration files on A3-B3.
6. This recovery is actually not necessary because the data in A2-B2 is perfectly in-sync due
to the synchronous replication approach. Nonetheless, the recovery will take place
because there was no automation window in place to freeze the configuration, which
would have removed the necessary paths between the local and the remote sites, thus
ensuring that no further I/Os continue to take place.
This does not happen with automated solutions such as GDPS (System z environments) and
eRCMF (open systems environments) that make use of the freeze capabilities of the IBM
System Storage DS8000. Had one of those automation softwares been there, after failing
over to the remote site, the database subsystem would have just restarted without the
necessity for a lengthy database recovery.
Figure 20-3 illustrates a rolling disaster situation where we have an automation software that
makes use of the freeze capability of the DS8000.
Database Database
7
Subsystem Subsystem
Switch over to remote site
1 6 Recover A2
Restart
source Hold I/O target
Storage Disk 3 Storage Disk 8
8 8
A
2C00
4 3C00
A1
A
Primary
Primary Synchronous B1
Primary
Log
Primary 1 Log'
Redrive Replicate
I/O
Storage Disk 5 Storage Disk

2D00 2 4
A
2D00 3D00
A
A
Primary
A2
Primary
Primary
Primary
Primary
Synchronous
B2
Primary
Primary
Primary Replicate Primary
DB 4 DB'

Subsystem 5 4 Subsystem 6
2D00
A
2E00
Synchronous A
3E00
A
Primary
A3
Primary
Primary
Primary
Primary Replicate B3
Primary
Primary
Primary Primary
RECON RECON'
Figure 20-3 Synchronous data replication, freeze, and restart without recovery required
The sequence of events in Figure 20-3 is the following:

1. Update intent to logging file.
2. The link between A2 and B2 becomes unavailable, or the storage disk subsystem with A2
fails, and a rolling disaster develops.
3. The first attempt to write to the database volume A2 will trigger a queue full condition. This
is also called an automation window.
4. Within this automation window, a freeze operation removes all related paths between both
sites for the selected LSSs. From now on, no data is replicated any longer between the
selected LSSs.
5. After the queue full condition expires, or an un-freeze operation is issued to end the freeze
period, the I/O is re-driven, and it may successfully complete when the pair is suspended,
or may fail when the storage disk subsystem 3 is not available any longer.
6. When the I/O fails, then a recovery required is written to the database configuration file on
A3. Note that volume A3 is no longer replicated due to the previous freeze. Note also that
volumes A2 and B2 are still identical and at the very same level of data currency — that of
the moment previous to the error.
7. The local site eventually becomes unavailable and a switchover to the recovery site is
carried out.
8. Database subsystem restart discovers a no recovery required indication because all
related data is consistent — the recovery required indication in step 6 was transmitted to
B3. Then the database subsystems continue immediately after the restart phase is
finished.

Summary
In summary, the following characteristics are typical of a synchronous data replication
technique:
򐂰 Application write I/O response time is affected. This can be modeled and predicted.
򐂰 Local and remote copies of data are committed to both storage disk subsystems before
host write I/O is complete.
򐂰 Data consistency is always maintained at the remote site as long as no failures occur. If a
rolling disaster occurs, freeze/run is needed to maintain consistency.
򐂰 Bandwidth between both sites has to scale with the peak write I/O rate.
򐂰 Data at the remote site is always current.
򐂰 No extra means, such as additional journal volumes or tertiary copy, are required.
򐂰 A tier 7 solution is achieved with automation software.
This is different with an asynchronous data replication approach. We still require data
consistency at a distant site in order to just restart at the distant site when the local site
becomes unavailable.
20.1.2 Asynchronous data replication and dependent writes

In normal operations, for asynchronous data replication, data consistency for dependent
writes will be preserved depending on the technique used to replicate the data. Dependent
writes and data consistency are explained in detail in 12.4, “Consistency Group function” on
page 116. For example, Global Copy, which is explained in Part 5, “Global Copy” on
page 185, and Global Mirror, that we are explaining now, use different techniques.
An asynchronous remote copy approach is usually required when the distance between the
local site and the remote site is beyond an efficient distance for a synchronous remote copy
solution. Metro Mirror provides an efficient synchronous approach for up to 300 km, when
utilizing Fibre Channel links.
Host
Server
1
2 Storage Disk
Storage Disk
2C00
A
A1
A
Primary
Primary Asynchronous B1
Primary
Primary 3 Replicate
source 4 target
local remote
Figure 20-4 Asynchronous data replication
In an asynchronous data replication environment, an application write I/O goes through the
following steps; see Figure 20-4:
1. Write application data to the source storage disk subsystem cache.
2. Present successful I/O completion to the host server. The application can then
immediately schedule the next I/O.
3. Replicate the data from the source storage disk subsystem cache to the target storage
disk subsystem cache.
4. Acknowledge to the source storage disk subsystem that data successfully arrived at the
target storage disk subsystem.
From the previous procedure, note how in an asynchronous type technique, the data
transmission and the I/O completion acknowledge are independent processes. This results in
virtually no application I/O impact, or at most to a minimal degree only. This also derives its
convenience when needing to replicate over long distances.
Global Copy non synchronous technique

On its own, a non synchronous technique like that of Global Copy provides no guarantee that
the sequence of arrival of application write I/Os to the source volumes is preserved at the
remote site. In other words, the order of dependent writes is not preserved at the remote site.
This is illustrated in Figure 20-5.
Host
Server
a b
1 2
Storage Disk
Subsystem 2
A
2C00
aPrimary
A
b Primary
Primary
Primary
Non synchronous
Replicate
Storage Disk 3 b
Subsystem 1
4
source target
local site
a in transit remote site
Figure 20-5 Global Copy sequence of data arrival not preserved at remote site
Global Copy by itself as a non synchronous data replication method does not provide data
consistency at the remote site.
In Figure 20-5 the sequence of data arrival at the local site is record b after record a. Due to
the way Global Copy cycles through its out-of-sync bit map, it may replicate record a to the
remote site after record b. This assumes that, for example, record a is written to a source disk
location that the Global Copy replication cycle just passed before. Global Copy may approach
another disk location that experienced just before an update with record b. Then record b gets
replicated to the remote site before the replication cycle starts from the beginning and finds
record a.

This situation may get even worse when involving multiple storage disk subsystems at the
local site. As Figure 20-6 shows, when using Global Copy, dependent writes are not
preserved at the recovery site when a failure occurs. This characteristic happens both at a
particular storage disk subsystem pair as well as across multiple storage disk subsystems.
Database
subsystem
local site remote site
source target
1 3
A
2C00
2C00 3C00
non synchronous
A
Primary
Primary
Primary
B
Primary
Log Log'
3 1
Replicate
Storage Disk 2 Storage Disk
2D00
A
A
2D00 non synchronous A
3D00
A
Primary
Primary
Primary
Primary
Primary B
Primary
Primary
Primary Primary
DB DB'
Replicate
Figure 20-6 Global Copy non synchronous data replication involving multiple disk subsystems
Notice that the numbers in Figure 20-6 indicate the transfer of data as well as the sequence of
its corresponding write I/Os. Record 3 may be replicated to storage disk subsystem 2 before
record 1 arrives to storage disk subsystem 2. Record 2 may never make it to storage disk
subsystem 4 due to the connectivity problem between storage disk subsystem 3 and storage
disk subsystem 4, which in turn may be the beginning of a rolling disaster. So, when using a
non-synchronous technique like Global Copy, consistent data cannot be guaranteed at the
remote site without any additional measures, functions, and procedures.
The challenge is to provide data consistency at the distant site at any time and independent of
the combination of storage disk subsystems involved. One solution is to combine Global Copy
with FlashCopy and create a three-copy solution. This requires a series of additional steps
that have to be organized and carried out:
1. At the local site, temporarily pause the application write I/Os on the source A volumes.
2. Wait for and make sure that the source A volumes and the target B volumes become
synchronized. Note the challenge to manage all volumes.
3. Using FlashCopy, create a point-in-time (PiT) copy of the B volumes at the remote site.
4. The FlashCopy targets, that is, the C volumes, will then hold a copy of the A volumes at
the time the application was paused or stopped. In this manner data consistency is kept at
the remote site.
5. Restart or resume the application write I/O activity to the A volumes.
Global Mirror asynchronous technique

If we could incorporate the previous five basic steps into the storage disk subsystem internal
microcode, and we could count with the corresponding management interface, then this
would basically be an efficient asynchronous mirroring technique that would allow the
replication of data over long distances, without impacting the application I/O response time,
its operation would be transparent and autonomic from the users point of view, and most
importantly would provide a consistent copy of the data at the remote site at all times. All this
is what Global Mirror is about.
To accomplish the necessary activities with minimum impact on the application write I/O,
Global Mirror introduces a smart bitmap approach in the source storage disk subsystem. With
this, Global Mirror can resume the application I/O processing immediately after a very brief
serialization period for all involved source storage disk subsystems. This brief serialization
periodically occurs at the very beginning of a sequence of events that resemble the ones
outlined above. In the following chapters we explain in detail the characteristics of Global
Mirror, how it operates, and how can you manage it.
Summary
In summary, the following characteristics are what an asynchronous data replication
technique should provide:
򐂰 Data replication to the remote site independent from application write I/O processing at the
local site. This derives in no impact, or at least only minimal impact, to the application local
write I/O response time.
򐂰 Data consistency and dependent writes always maintained at the remote site.
򐂰 Data currency at the remote site lags a little behind the local site. The remote site is
always less current than the local site. In peak write workloads this difference is going to
increase. Global Mirror does not throttle host write I/Os to eventually manage this
discrepancy.
򐂰 The bandwidth requirement between local and remote sites does not have to be
configured for peak write workload. Link bandwidth utilization is improved over
synchronous solutions.
򐂰 Tertiary copies are required at the remote site to preserve data consistency.
򐂰 Data loss in disaster recovery situations is limited to the data in transit plus the data that
may still be in the queue at the local site waiting to be replicated to the recovery site.
All the previous attributes are characteristics of Global Mirror.

20.2 Basic concepts of Global Mirror
It is important to understand that Global Mirror works like a distributed application. A
distributed application is usually built on a server to client relationship. The server functions
as a supervisor and instructs the client. The client is able to do some work in an autonomic
fashion but relies on the coordination efforts from the server; see Figure 20-7.
Task1 Local site Remote site

Task2
Task1
Server
Task3 Client
Network
Task3
Task2
Client
Client
Figure 20-7 Distributed application
The server distributes the work to its clients. The server also coordinates all individual
feedback from the clients and decides based on this feedback further actions.
Looking at Figure 20-7, it is obvious that the communication paths between the server and all
its clients are key. Without communication paths between these four components the
functions will eventually come to a complete stop. Matters get more complicated when the
communication fails unexpectedly in the middle of information exchange between the server
and his clients or to some of his clients.
Usually a two-phase commit process helps to provide a consistent state for certain functions
and whether they have successfully completed at the client site. Once a function is
successfully completed and is acknowledged to the server, the server progresses to the next
function task. At the same time the server tries to do as much as possible in parallel to
minimize the hit on throughput due to serialization and checkpoints.
When certain activities are dependent on each other it is required that the server coordinate
these activities to ensure a proper sequence.
The server and client can be replaced by terms such as master and subordinate; see
Figure 20-8. These terms are used later when discussing Global Mirror.

FlashCopy 1
FlashCopy 1
Master
Subord Flash 2 Target
source
Network
FlashCopy2
FlashCopy2
Subordinate
Target
source
Figure 20-8 Global Mirror as a distributed application
Figure 20-8 shows the basic Global Mirror structure. A master coordinates all efforts within a
Global Mirror environment. Once the master is started and manages a Global Mirror
environment, the master issues all related commands over inband communication to its
attached subordinates at the local site. This may include a subordinate within the master
itself. This communication between the master and an internal subordinate is transparent and
does not need any extra attention from the user. The subordinates use inband communication
to communicate with their related target storage disk subsystems at the remote site. The
master also receives all acknowledgements from his subordinates and their targets, and
coordinates and serializes all the activities in the session.
When the master and subordinate are in a single storage disk subsystem the subordinate is
internally managed by the master. With two or more storage disk subsystems at the local site,
which participate in a Global Mirror session, the subordinate is external and needs separate
attention when creating and managing a Global Mirror session or environment.
The following sections explain how Global Mirror works and how Global Mirror ensures
consistent data at any time at the remote site. First we go through the process of how to
create a Global Mirror environment. At the same time, this explanatio, gives us a first insight
on how Global Mirror works.
20.3 Set up a Global Mirror session

Global Mirror, as a long-distance remote copy solution, is based on an efficient combination of
Global Copy and FlashCopy functions. It is the microcode that provides, from the user
perspective, a transparent and autonomic mechanism to intelligently utilize Global Copy in
conjunction with certain FlashCopy operations to attain consistent data at the remote site.
In order to understand how Global Mirror works, we start explaining first how a Global Mirror
environment, that is, a Global Mirror session, is created and started. This is a step-by-step
approach and helps to understand further the Global Mirror operational aspects.

20.3.1 Simple configuration to start
In order to understand each step and to show the principles, we start with a simple application
environment where a host makes write I/Os to a single application volume (A); see
Figure 20-9.
Host
Write I/O
A
Primary
A
Primary
Primary
Primary
Primary
A
Primary
Primary
Primary

Figure 20-9 Start with simple application environment
20.3.2 Establish connectivity to remote site

Now we add a distant site, which has a storage disk subsystem (B), and we want to
interconnect both sites; see Figure 20-10.
Host
Write I/O
A
Primary
A
Primary
Primary
Global Copy path
Primary
B
Primary
A
Primary
Primary
Primary
Local site FCP port Remote site

Figure 20-10 Establish Global Copy connectivity between both sites
Note in Figure 20-10 that we establish Global Copy paths over an existing network. This
network may be based on an FCP transport technology or on an IP-based network.
Global Copy paths are logical connections that are defined over the physical links that
interconnect both sites. Note that all remote mirror and copy paths (that is Metro Mirror,
Global Copy, and Global Mirror paths) are similar and are based on FCP. The term Global
Copy path just denotes that the path is intended for Global Copy use.
20.3.3 Create Global Copy relationships
Host
Write I/O
A
Primary
A
Primary
Primary
Primary
B
Primary
A
Primary Global Copy
target
source Primary
copy pending Primary
copy pending
O
O
S OOS: Out-of-sync bit map

Figure 20-11 Establish Global Copy volume pair
Next, we create a Global Copy relationship between the source volume and the target
volume; see Figure 20-11. This step changes the target volume state from simplex (no
relationship) to target copy pending. This copy pending state applies to both volumes, source
copy pending and target copy pending. At the same time data starts to be copied from the
source volume to the target volume. After a first complete pass through the entire A volume,
Global Copy will constantly scan through the out-of-sync bit map. This bitmap indicates
changed data as it arrives from the applications to the source disk subsystem. Global Copy
replicates the data from the A volume to the B volume based on this out-of-sync bit map.
In the following paragraphs we refer to the source volume as the A volume and to the target
volume as the B volume for simplicity.
Global Copy does not immediately copy the data as it arrives to the A volume. Instead, this is
an asynchronous process. As soon as a track is changed by an application write I/O, it is
reflected in the out-of-sync bitmap as with all the other changed tracks. There can be several
concurrent replication processes that work through this bitmap, thus maximizing the utilization
of the high bandwidth Fibre Channel links.
This replication process keeps running until the Global Copy volume pair A-B is explicitly or
implicitly suspended or terminated. Note that a Global Mirror session command, for example,
to pause or to terminate a Global Mirror session, does not affect the Global Copy operation
between both volumes.
At this point data consistency does not yet exist at the remote site.

20.3.4 Introduce FlashCopy
FlashCopy is an integral part of the Global Mirror solution, and now it follows as the next step
in the course of establishing a Global Mirror session; see Figure 20-12.
Host
Write I/O
FlashCopy
A
Primary
A
Primary
Primary
Primary
B
Primary
A
Primary Global Copy
target
source Primary
Primary
copy pending
copy pending C
tertiary
O
O SBM: Source Bit Map S T
S B B
TBM: Target Bit Map M M

Figure 20-12 Introducing FlashCopy in the Global MIrror solution
The focus is now on the remote site. Figure 20-12 shows a FlashCopy relationship with a
Global Copy target volume as the FlashCopy source volume. Volume B is now both at the
same time a Global Copy target volume and a FlashCopy source volume. In the same disk
subsystem is the corresponding FlashCopy target volume.
Note that this FlashCopy relationship has certain attributes that are typical and required when
creating a Global Mirror session. These attributes are:
򐂰 Inhibit target write: Protect the FlashCopy target volume from being modified by anyone
other than Global Mirror related actions.
򐂰 Enable change recording: Apply changes only from the source volume to the target
volume that occurred to the source volume in between FlashCopy establish operations,
except for the first time when FlashCopy is initially established.
򐂰 Make relationship persistent: Keep the FlashCopy relationship until explicitly or implicitly
terminated. This parameter is automatic due to the nocopy property.
򐂰 Nocopy: Do not initiate background copy from source to target, but keep the set of
FlashCopy bitmaps required for tracking the source and target volumes. These bitmaps
are established at the first time when a FlashCopy relationship is created with the nocopy
attribute. Before a track in the source volume B is modified, between Consistency Group
creations, the track is copied to the target volume C to preserve the previous point-in-time
copy. This includes updates to the corresponding bitmaps to reflect the new location of the
track that belongs to the point-in-time copy. Note that each Global Copy write to its target
volume within the window of two adjacent Consistency Groups may cause FlashCopy I/O
operations.
20.3.5 Define the Global Mirror session
Creating a Global Mirror session does not involve any volume within the local nor the remote
sites. Our focus is back onto the local site again; see Figure 20-13.
Host
Define Global Mirror session
FlashCopy
01 A
Primary
B
Primary
Global Copy
target
Primary
Primary
copy pending
C
tertiary
S T
B B
M M
Figure 20-13 Define Global Mirror session
Defining a Global Mirror session creates a kind of token, which is a number between 1 and
255. This number represents the Global Mirror session.
This session number is defined at the LSS level. Each LSS that has volumes that will be part
of the session needs a corresponding define session command. Currently only a single
session is allowed per storage disk subsystem. The architecture allows for more than one
session and will be exploited in the future.

20.3.6 Populate the Global Mirror session with volumes
The next step is the definition of volumes in the Global Mirror session. The focus is still on the
local site; see Figure 20-14. Note that only Global Copy source volumes are meaningful
candidates to become a member of a Global Mirror session.
Host
Add Global Copy source volumes to Global Mirror session
FlashCopy
01
A
A
Primary
Primary
B
Primary
A
Primary Global Copy
target
source Primary
copy pending Primary
copy pending C
tertiary
O
O S T
S B B
M M

Figure 20-14 Add Global Copy source volumes to Global Mirror session
This process adds source volumes to a list of volumes in the Global Mirror session. But at this
stage it still does not perform consistency group formation.
Note that Global Copy is replicating, on the Global Copy target volumes, the application
updates that arrive to the Global Copy source volumes. Initially the Global Copy source
volumes are placed in a join pending state. Once a Consistency Group is formed, the Global
Copy source volume will then be added to the session and will be placed in an in session
state.
Nothing happens to the C volume after its initial establishment in a Global Mirror session.
20.3.7 Start the Global Mirror session
This is the last step and it starts the Global Mirror session. Upon this, Global Mirror starts to
form Consistency Groups at the remote site. As Figure 20-15 indicates, the focus here is on
the local site, with the start command issued to an LSS in the source storage disk subsystem.
With this start command you set the master storage disk subsystem and the master LSS.
From now on, session related commands have to go through this master LSS.
Host
Start Global Mirror session
FlashCopy
01
A
A
Primary
Primary
B
Primary
A
Primary Global Copy
target
source Primary
Primary
copy pending
copy pending C
tertiary
O
C
O S C T
R
S CR: Change recording bit map B R B
M M

Figure 20-15 Start Global Mirror
This start command triggers events that involve all the volumes within the session. This
includes a very fast bitmap management at the local storage disk subsystem, issuing inband
FlashCopy commands from the local site to the remote site, and verifying that the
corresponding FlashCopy operations successfully finished. This all happens at the microcode
level of the related storage disk subsystems that are part of the session, fully transparently,
and in an autonomic fashion from the users’ perspective.
All C volumes that belong to the Global Mirror session comprise the Consistency Group. Now
let us see more details about the Consistency Group creation at the remote site.
20.4 Consistency Groups

To achieve the goal of creating a set of volumes at a remote site that contains consistent data,
asynchronous data replication alone is not enough — it must be complemented with either a
kind of journal or a tertiary copy of the target volume. With Global Mirror this third copy is
naturally created through the use of FlashCopy.
The microcode automatically triggers a sequence of autonomic events to create a set of

consistent data volumes at the remote site. We call this set of consistent data volumes a
Consistency Group. The following sections describe the sequence of events that create a
Consistency Group.

20.4.1 Consistency Group formation
The creation of a Consistency Group requires three steps that are internally processed and
controlled by the microcode. Outside of the Licensed Internal Code (LIC), these steps are
fully transparent and do not require any other external code invocation or user action.
The numbers in Figure 20-16 illustrate the sequence of the events involved in the creation of
a Consistency Group. This illustration provides a high-level view that is sufficient to
understand how this process works.
Done Done Done

Start Start Start
Serialize all
1 2 3 Perform
Global Copy Drain data from local to remote site
FlashCopy
source volumes
O
C A1 O B1
R source S target C1
tertiary
O
C
R A2 O B2
source S target C2
tertiary
Local Remote
Figure 20-16 Formation of consistent set of volumes at the remote site
Note that before step 1 and after step 3, Global Copy constantly scans through the
out-of-sync bitmaps and replicates data from A volumes to B volumes as described in 20.3.3,
“Create Global Copy relationships” on page 255.
When the creation of Consistency Group is triggered, the following steps occur in a serial
fashion:
1. Serialize all Global Copy source volumes. This imposes a brief hold on all incoming write
I/Os to all involved Global Copy source volumes. Once all source volumes are serialized,
the pause on the incoming write I/O is released and all further write I/Os are now noted in
the change recording bitmap. They are not replicated until step 3 is done, but application
write I/Os can immediately continue.
2. Drain includes the process to replicate all remaining data that is indicated in the
out-of-sync bitmap and still not replicated. Once all out-of-sync bitmaps are empty (note
that empty here does not mean in a literal sense) step 3 is triggered by the microcode from
the local site.
3. Now the B volumes contain all data as a quasi point-in-time copy, and are consistent due
to the serialization process in step 1 and the completed replication or drain process in step
2. Step 3 is now a FlashCopy that is triggered by the local system’s microcode as an
inband FlashCopy command to volume B, as FlashCopy source, and volume C, as
FlashCopy target volume. Note that this FlashCopy is a two-phase process: First, the
FlashCopy command to all involved FlashCopy pairs in the Global Mirror session. Then
the master collects the feedback and all incoming FlashCopy completion messages. When
all FlashCopy operations are successfully completed, then the master concludes that a
new Consistency Group has been successfully created.
FlashCopy applies here only to changed data since the last FlashCopy operation. This is
because the start change recording property was set at the time when the FlashCopy
relationship was established. The FlashCopy relationship does not end due to the nocopy
property, which is also assigned at FlashCopy establish time; see 20.3.4, “Introduce
FlashCopy” on page 256. Note that the nocopy attribute results in that no physical
background copy is done for any tracks. Only bitmaps are maintained and updated.
Once step 3 is complete, a consistent set of volumes have been created at the remote. This
set of volumes, the C volumes, represents the Consistency Group.
At this very moment also, the B volumes and the C volumes are, only for this brief moment,
equal in their content. Immediately after the FlashCopy process is logically complete, the local
systems’s microcode is notified to continue with the Global Copy process. In order to replicate
the changes to the A volumes that occurred during the step 1 to step 3 window, the change
recording bitmap is mapped against the empty out-of-sync bitmap, and from now on, all
arriving write I/Os will end up again in the out-of-sync bitmap. From now on the conventional
Global Copy process, as outlined in 20.3.3, “Create Global Copy relationships” on page 255,
continues until the next Consistency Group creation process is started.
20.4.2 Consistency Group parameters

In the previous section we described the steps followed during the creation of a Consistency
Group. These steps can be adjusted by means of the tuning parameters of Global Mirror.
Maximum Maximum
coordination drain
time time
Serialize all
1 2 3 Perform
Global Copy Drain data from local to remote site
FlashCopy
souce volumes
CG interval
1 2 3 time
1 2 3
Figure 20-17 Consistency Group tuning parameters
Global Mirror provides a set of three externalized parameters that can be used for tuning the
Consistency Group formation process, overriding the default values; see Figure 20-17:
򐂰 Maximum coordination time: dictates, for the Global Copy source volumes that belong to
this Consistency Group, how long a host write I/O may be impacted due to coordination
and serialization activities. This time is measured in milli-seconds (ms). The default is
50 ms.
򐂰 Maximum drain time: is the maximum time allowed for draining the out-of-sync bitmap
once the process to form a Consistency Group is started and step 1 successfully
completed. The maximum drain time is specified in units of seconds. The default is
30 seconds.
If the maximum drain time is exceeded, Global Mirror will fail to form the Consistency
Group and will evaluate the current throughput of the environment. If this indicates that
another drain failure is likely, Global Mirror will stay in Global Copy mode while regularly

re-evaluating the situation to determine when to start to form the next Consistency Group.
If this persists for a significant period of time, then eventually Global Mirror will force the
formation of a new Consistency Group. In this way Global Mirror ensures that during
periods when the bandwidth is insufficient, production performance is protected and data
is transmitted to the remote site in the most efficient manner possible. When the peak
activity has passed, consistency group formation will resume in a timely fashion.
򐂰 Consistency Group interval time: once a Consistency Group has been created, the CG
interval time determines how long to wait before starting with the formation of the next
Consistency Group. This is specified in seconds, and the default is zero (0) seconds. Zero
seconds means that Consistency Group formation happens constantly. As soon as a
Consistency Group is successfully created, the process to create a new Consistency
Group starts again immediately.
There is no external parameter to limit the time for the FlashCopy operation. This is due to its
very fast bitmap manipulation process.
In the first step, the serialization step, when the Consistency Group spans over more than
one source disk subsystem, Global Mirror not only serializes all related Global Mirror source
volumes but also does the coordination with other storage disk subsystems.
Global Mirror utilizes a distributed approach as well as a two-phase commit technique for
activities between the master and its subordinate LSSs. The communication between the
local and the remote site is organized through the subordinate LSS. The subordinates
function here partly as a transient component for the Global Mirror activities, which are all
triggered and coordinated by the master. This distributed concept allows ways to provide a set
of data consistent volumes at the remote site independent of the number of involved storage
disk subsystems at the local or at the remote site.
21
Chapter 21. Global Mirror options and

configuration
In this chapter we provide a detailed description of Global Mirror options, including how to
create a Global Mirror environment and how to remove it. We discuss how to change Global
Mirror tuning parameters and how to modify an active Global Mirror session. We also discuss
a scenario where a site switch is performed due to the local site failure.
Global Mirror is intended for long distance data replication and, for this, it relies on the
network infrastructure and components used between the remote sites. This chapter does not
cover network-related topics. For information about these topics, refer to the IBM Redbook
IBM TotalStorage Business Continuity Solutions Guide, SG24-6547.

21.1 Terminology used in Global Mirror environments
First, let us review and further define some of the terms and new elements we have
presented so far, and that are of common use when working in a Global Mirror context.
Dependent writes
If the start of one write operation is dependent upon the completion of a previous write, the
writes are dependent. Application examples for dependent writes are databases with their
associated logging files. For instance, the database logging file will be updated after a new
entry has been successfully written to a tablespace.
Compliance at the remote site, with the chronological order of dependent writes to source
volumes, is the basis to provide consistent data at the remote site through remote copy
operations.
Dependent writes is discussed in detail in 12.4.1, “Data consistency and dependent writes”
on page 117, in the Metro Mirror part.
Consistency
The consistency of data is ensured if the order of dependent writes to disks or disk groups is
maintained. With Copy Services solutions the data consistency at the remote site is important
for the usability of the data. Consistent data, for instance, gives the ability to perform a data
base restart rather than a data base recovery that could take hours or even days.
Data consistency across all target volumes spread across multiple storage disk subsystems
is essential for logical data integrity.
Data currency
This term describes the difference of time since the last data was written at the production
site, versus the time the same data was written to the remote site. It determines the amount of
data you have to recover at the remote site after a disaster. This is also called recovery point
objective or RPO. Only synchronous copy solutions such as Metro Mirror have a currency of
zero or RPO = zero. All asynchronous copy solutions have a data currency greater than zero.
With Global Mirror a data currency of a few seconds can be achieved, while data consistency
is always maintained by the Consistency Group process within Global Mirror.
For asynchronous replication, this means that the data is not replicated at the same time as
the local I/O happens, but the data is replicated with certain time lag. Examples of different
non-synchronous or asynchronous replication methods are:
򐂰 Global Copy is a non-synchronous method that does not guarantee consistent data at the
remote site.
򐂰 z/OS Global Mirror (formerly XRC) is an asynchronous replication method that guarantees
consistent data at the remote site.
򐂰 Global Mirror is also an asynchronous replication method that provides consistent data at
the remote site.
Session
A Global Mirror session is a collection of volumes that are managed together when creating
consistent copies of data volumes. This set of volumes can reside in one or more LSSs and
one or more storage disk subsystems at the local site. Open systems volumes and z/OS
volumes can both be members of the same session.
When you start or resume a session, the creation of Consistency Groups is performed, and
the master storage disk subsystem controls the session by communicating with the
subordinate storage disk subsystems.
There is also a session concept at the LSS level. But all LSS sessions are combined and
grouped together within a Global Mirror session.
Master
The master is a function inside a source storage disk subsystem that communicates with
subordinates in other storage disk subsystems, and controls the creation of Consistency
Groups while managing the Global Mirror session. The master is defined when the start
command for a session is issued to any LSS in a source storage disk subsystem. This
determines who becomes the master storage disk subsystem.
The master needs communication paths over Fibre Channel links to any one of the LSSs in
each subordinate disk storage subsystem.
Subordinate
The subordinate is a function inside a source storage disk subsystem that communicates with
the master and is controlled by the master. At least one of the LSSs of each subordinate
source storage disk subsystems needs Fibre Channel communication paths to the master.
This enables the communication between the master and the subordinate, and is required to
create Consistency Groups of volumes that spread more than one storage disk subsystem.
If all the volumes of a Global Mirror session reside in one source storage disk subsystem no
subordinate is required, because the master can communicate to all LSSs inside the source
storage disk subsystem.
Consistency Group
This is a group of volumes in one or more target storage disk subsystems whose data must
be kept consistent at the remote site.
Local Site
This is the site that contains the production servers. This and other publications use this term
interchangeably with the terms primary, production, or source site.
Remote Site
This is the site that contains the backup servers of a disaster recovery solution. This and
other publications use this term interchangeably with the terms secondary, backup, standby,
or target site.
21.2 Create a Global Mirror environment

The following section recommends a certain sequence of steps to establish a Global Mirror
environment. This is independent of the interface used to create a Global Mirror environment.
Note that for illustration purposes, parameters and keywords are used in this section that
apply to the DS CLI commands.
Chapter 21. Global Mirror options and configuration 265

We explain a basic Global Mirror setup, illustrated in Figure 21-1.
Global Copy
01 A
Network
A Primary
B
Primary
Primary
Subordinate A
Primary
target Primary
Primary
source
copy pending
copy pending C
tertiary
paths
Local site Remote site(s)
01 A
A Primary
B
Primary
Master Primary
A
Primary
target Primary
Network Primary
source
copy pending
copy pending C
tertiary
Global Copy
Figure 21-1 Global Mirror basic configuration with master and subordinate disk subsystems
The order of commands to create a Global Mirror environment is not completely fixed and
allows for some variation. In order to be consistent with other sources and to not confuse the
user with a different sequence of commands, we recommend a meaningful order and suggest
the following steps to create a Global Mirror environment:
1. Define the paths between the local site and the remote site. In Figure 21-1, these are the
logical communication paths between corresponding LSSs at the local site and the remote
site, defined over Fibre Channel physical links that are configured over the network. Global
Copy source LSSs are represented by the A volumes and their corresponding Global
Copy target LSSs by the B volumes.
You may also define logical communication paths here between the master and any
subordinate storage disk subsystem that will be part of the Global Mirror session. Note
that these paths are defined between source storage disk subsystems at the local site.
With only a single source storage disk subsystem, you do not need to define paths to
connect internal LSSs within the source storage disk subsystem. The communication
between the master and the subordinates within a single source storage disk subsystem is
transparent and internally performed.
2. Once the communication paths are defined, start the Global Copy pairs that will be part of
a Global Mirror session. Global Copy has to be created with the mkpprc -type gcp
command. We recommend that you wait until the first initial copy is complete before you
continue to the next step. This avoids unnecessary FlashCopy background I/Os in the
following step.
3. The next step is to create FlashCopy relationships between the B and C volumes. You can
use the mkremoteflash (abbreviated mkflash) command, with the following parameters:
-tgtinhibit, -record, and -nocp. The -persist parameter is automatically set when the
-record parameter is selected. For a discussion of the particular FlashCopy attributes that
are required for a Global Mirror FlashCopy see 20.3.4, “Introduce FlashCopy” on
page 256.
4. With external subordinates, that is, with more than one involved disk subsystem at the
local site, you need paths between the master LSS and any potential subordinate storage
disk subsystem at the local site. If you did not define these paths in the very first step, then
this is the time to create these paths before you continue with the next step.
5. Define a token that identifies the Global Mirror session. This is a session ID with a number
between 1 and 255. Define this session number to the master storage disk subsystem and
also to all potentially involved source LSSs that are going to be part of this Global Mirror
session and will contain Global Copy source volumes that will belong to the Global Mirror
session. All source LSSs include all LSSs in potential subordinate storage disk
subsystems that are going to be part of the Global Mirror session. For this step you can
use the mksession command.
6. The next step populates the session with the Global Copy source volumes. You should put
these Global Copy source volumes into the session after they complete their first pass for
initial copy. For this step you can use the chsession -action add -volume command.
7. Start the session. This command actually defines the master LSS. All further session
commands have to go through this LSS. You may specify with the start command the
Global Mirror tuning parameters such as maximum drain time, maximum coordination
time, and Consistency Group interval time. For this step you can use the mkgmir
command.
You go through this recommended sequence independent of the interface used.
21.3 Modify a Global Mirror session

Once a session is active and running, you may alter the Global Mirror environment to add or
remove volumes. You also may add storage disk subsystems to a Global Mirror session or
you may change the interval between the formation of Consistency Groups.
21.3.1 Add or remove volumes to the Global Mirror session

Volumes can be added to the session at any time after the session number is defined to the
LSS where the volumes reside. Once the session is started, volumes can be added to the
session or can be removed from the session also at any time.
Note: You only add Global Copy source volumes to a Global Mirror session.
Volumes may be added to a session in any state, for example, simplex or pending. Volumes
that have not completed their initial copy phase stay in a join pending state until the first initial
copy is complete. If a volume in a session is suspended, it will cause Consistency Group
formation to fail.
We recommend that you add only Global Copy source volumes that have completed their
initial copy or first pass, although the microcode itself will stop volumes from joining the Global
Mirror session until the first pass is complete. Also, we recommend that you wait until the first
initial copy is complete before you create the FlashCopy relationship between the B and the C
volumes.

Note: You cannot add a Metro Mirror source volume to a Global Mirror session. Global
Mirror supports only Global Copy pairs. When Global Mirror detects a volume that, for
example, is converted from Global Copy to Metro Mirror, the following formation of a
Consistency Group will fail.
When you add a rather large number of volumes at once to an existing Global Mirror session,
then the available resources for Global Copy within the affected ranks may be utilized by the
initial copy pass. To minimize the impact to the production servers when adding many new
volumes, you may consider adding the new volumes to an existing Global Mirror session in
stages.
Suspending a Global Copy pair that belongs to an active Global Mirror session will impact the
formation of Consistency Groups. When you intend to remove Global Copy volumes from an
active Global Mirror session, follow these steps:
1. Remove the desired volumes from the Global Mirror session.
2. Withdraw the FlashCopy relationship between the B and C volumes.
3. Terminate the Global Copy pair to bring volume A and volume B into simplex mode.
Note: When you remove A volumes without pausing Global Mirror, you might see this
reflected as an error condition with the showmigr -metrics command, indicating that the
Consistency Group formation failed. However, this does not mean you have lost a
consistent copy at the remote site because Global Mirror does not take the FlashCopy (B
to C) for the failed Consistency Group data. This message indicates that just one
Consistency Group formation has failed, and Global Mirror will retry the sequence.
21.3.2 Add or remove storage disk subsystems or LSSs

When you plan to add a new subordinate storage disk subsystem to an active session, you
have to stop the session first. Then add the new subordinate storage disk subsystem and
start the session again. The session start command will then contain the new subordinate
storage disk subsystem. The same procedure applies when you remove a storage disk
subsystem from a Global Mirror session, which can be a subordinate only. In other words,
you cannot remove the master storage disk subsystem.
When you add a new LSS to an active session and this LSS belongs to a storage disk
subsystem that already has another LSS that belongs to this Global Mirror session, you may
add the new LSS to the session without stopping and starting the session again. This is true
for either the master or for a subordinate storage disk subsystem.
21.3.3 Modify the Global Mirror session parameters

The parameters that are used for tuning a Global Mirror session may be modified by pausing
the session and then resuming the session with the new values. The parameters that you may
modify are:
򐂰 Consistency Group interval time
򐂰 Maximum coordination interval time
򐂰 Maximum Consistency drain time
When you give a start command after a pause command, as opposed to a resume, any value
for the Consistency Group interval time, maximum coordination interval time, or maximum
Consistency Group drain time in the start command is ignored. If these parameters have to
be altered, you have to give a resume command to the paused session with the parameters
specified. If you resume a paused session without specifying these parameters, they will be
set to their default values; see 20.4.2, “Consistency Group parameters” on page 261.
Important: When setting new values for the tuning parameters, be sure to check for errors
in Consistency Group formation and in draining the out-of-sync bitmaps. A few errors are
not significant and do not jeopardize the consistency of your Global Mirror. However, if
failures repeatedly occur, for example, no more consistency groups are formed, or the
percentage of successful Consistency Groups is unacceptable, or the frequency of
Consistency Groups is not meeting your requirements (Recovery Point Objective-RPO),
then the new values set for the tuning parameters need to be revised and changed.
21.3.4 Global Mirror environment topology changes

The topology of a Global Mirror session, for example, the master SSID, subordinate SSIDs,
and subordinate serial numbers, cannot be altered through a pause and resume command
sequence. When you resume the session after a pause and the Global Mirror topology is not
the same as it was at the pause, the start of the session will fail.
If you need to change the topology of your Global Mirror session, you have to stop the session
and start the session again with the new topology structure information in the mkgmir
command.
Topology also refers to the list of storage disk subsystems that are subordinates. You define
control paths between the master and subordinate LSSs. One LSS per subordinate disk
subsystem is sufficient. When you define the control path you identify the source LSS on the
master disk subsystem. The target LSS in the path definition command points to a
corresponding subordinate disk subsystem. These LSSs go into the topology specification
that defines the communication paths between master and subordinate storage disk
subsystems. To change these values you must stop the Global Mirror process (rmgmir
command) and start it again with the new topology specifications (mkgmir command).
21.3.5 Remove FlashCopy relationships

When you withdraw a FlashCopy relationship within a Global Mirror session, the Consistency
Group process is affected. If a withdraw is required, then first remove the Global Copy source
volume from the session.
Once the volumes are removed from the session, you may explicitly terminate the
corresponding FlashCopy relationship that was tied to this source volume.
The termination of FlashCopy relationships, may be needed when you want to change the
FlashCopy targets within a Global Mirror configuration and choose, for example, another LSS
for the FlashCopy targets. This may be needed because you want to replace the FlashCopy
targets due to a skew in the load pattern in the remote storage disk subsystem. In this
situation you may pause the session before such activity, and then resume the session again
once the removal of the FlashCopy relationships is completed.
Note: A pause command (pausemigr) will complete a Consistency Group formation in

progress. A stop (rmgmir) will immediately end the formation of Consistency Groups.

21.3.6 Remove the Global Mirror environment
To remove a Global Mirror environment and remove all traces of Global Mirror we recommend
the following sequence of steps:
1. Terminate the Global Mirror session.
2. Remove all Global Copy source volumes from the Global Mirror session.
3. Close the Global Mirror session.
4. Withdraw all FlashCopy relationships between the B and C volumes.
5. Terminate all Global Copy pairs.
6. Remove the paths between the local and remote sites.
7. Remove the paths between the master LSS and all related subordinate LSSs.
21.4 Global Mirror with multiple storage disk subsystems

When you create a Global Mirror environment that spans multiple storage disk subsystems at
the local site, and probably also at the remote site, then you need to define communication
paths between the involved local storage disk subsystems.
Define Global Mirror session
01 Network A
B
Primary
Primary
Primary
Primary
C
Define Global Mirror session Remote site
01 A
Primary
B
Primary
Primary
Network Primary
C
Figure 21-2 Define Global Mirror session to all potentially involved storage disk subsystems
Figure 21-2 on page 270 shows a symmetrical configuration with a one-to-one mapping. You
have to define the corresponding session, with its number, to all potentially involved LSSs at
the local site. There is still no connection between both local storage disk subsystems.
Therefore we have to define the corresponding Global Mirror paths; see Figure 21-3.
Global Copy
01
Network A
A
Primary
Primary
B
Primary
Subordinate A
Primary
target Primary
Primary
source
copy pending
copy pending C
tertiary
Local site paths Remote site
Start Global Mirror session
01 A
A Primary
B
Primary
Master Primary
A
Primary
target Primary
Network Primary
source
copy pending
copy pending C
tertiary
Global Copy
Figure 21-3 Decide for master disk subsystem and start Global Mirror session
Through the start command mkgmir, you decide which LSS becomes the master LSS and
consequently which local storage disk subsystem becomes the master storage disk
subsystem. This master acts like a server in a client/server environment.
The required communication between the master storage disk subsystem and the
subordinate storage disk subsystems is inband, over the defined Global Mirror paths. This
communication is highly optimized, and minimizes any potential application write I/O impact
during the coordination phase to about a few milliseconds. For details see 20.4, “Consistency
Groups” on page 259.

Note that this communication is performed over FCP links. At least one FCP link is required
between the master storage disk subsystem and the subordinate storage disk subsystem.
Figure 21-4 uses dashed lines to show the Global Mirror paths that are defined over FCP
links, between the master storage disk subsystem and its associated subordinate storage
disk subsystems. These FCP ports are dedicated for Global Mirror communication between
master and subordinates.
01
A
A
Primary
Primary
B
Primary
Subordinate A
Primary
target Primary
Primary
source
copy pending
copy pending
Global Copy links C
tertiary
01
A
A
Primary
Primary
B
Primary
Subordinate A
Primary
target Primary
Primary
source
copy pending
copy pending
Global Copy links C
tertiary
GM links
01 A
A Network Primary
B
Primary
Master Primary
A
Primary
target Primary
Primary
source
copy pending Global Copy links
copy pending C
tertiary
Figure 21-4 Global Mirror paths over FCP links between source storage disk subsystems
Also shown in Figure 21-4 is a shared port on the master storage disk subsystem, and
dedicated ports at the subordinates. Not considering availability, from a communications and
traffic viewpoint only, a link would be sufficient for the traffic between the master and its
subordinates. For redundancy, we suggest configuring two links.
Note that when you configure links over a SAN network, the same FCP ports of the storage
disk subsystem may be used for the Global Mirror session communication, as well as for the
Global Copy communication, and for host connectivity. However, for performance reasons,
and to prevent host errors from disrupting your Global Mirror environment, it is often a good
idea to use separate FCP ports.
The sample configuration shown in Figure 21-5 shows a mix of dedicated and shared FCP
ports. In this example an FCP port in the master storage disk subsystem is used as Global
Mirror link to the other two subordinate storage disk subsystems, and is also used as Global
Copy link to the target disk subsystem. Also, there are ports at the subordinate disk
subsystem that are used as Global Mirror session link as well as Global Copy link.
01
A
A
Primary
Primary
B
Primary
Subordinate A
Primary
target Primary
Primary
source
copy pending
copy pending
Global Copy links C
tertiary
01
A
A
Primary
Primary
B
Primary
Subordinate A
Primary
target Primary
Primary
source
copy pending
copy pending Global Copy links C
tertiary
GM links
01 A
A Network Primary
B
Primary
Master Primary
A
Primary
target Primary
Primary
source
copy pending
C
tertiary
Figure 21-5 Dedicated and shared links

If possible, a better configuration is the one shown in Figure 21-6. Again, from a performance
and throughput viewpoint you would not need two Global Mirror links between the master and
its subordinate storage disk subsystems. Still, dedicated ports for Global Mirror control
communication between master and subordinate provides a maximum of responsiveness.
01
A
A Primary
B
Primary
Primary
Subordinate A
Primary
target Primary
Primary
source
copy pending
copy pending
Global Copy links C
tertiary
01
A
A
Primary
Primary
B
Primary
Subordinate A
Primary
target Primary
Primary
source
copy pending
copy pending Global Copy links C
tertiary
GM links
01 A
A Network Primary
B
Primary
Master Primary
A
Primary
target Primary
Primary
source
copy pending
C
tertiary
Figure 21-6 Dedicated Global Mirror links and dedicated Global Copy links
21.5 Recovery scenario after production site failure

This section covers the steps that you need to follow in a Global Mirror environment when a
production site failure requires you to recover at the remote site.
21.5.1 Normal Global Mirror operation
Figure 21-7 shows a simple configuration with Global Mirror active and running.
Server
FlashCopy
Global Copy A
A A
Primary
Primary
A
Primary
Primary
Primary
Primary
B
Primary
A
A
Primary
source
target
A
Primary
Primary
Primary
copy pending
copy pending C
Primary
tertiary

Figure 21-7 Normal Global Mirror operation
Writes from the server are replicated through Global Copy, and Consistency Groups are
created as tertiary copies. The B volumes are Global Copy target volumes, and they are also
FlashCopy source volumes. The C volumes are the FlashCopy target volumes. The
FlashCopy relationship is a particular relationship; see 20.3.4, “Introduce FlashCopy” on
page 256.
21.5.2 Production site failure

A failure at the local site prevents all I/O to and from the local storage disk subsystems; see
Figure 21-8 on page 276. This may have some impact on the formation of Consistency
Groups because the entire process is managed and controlled by the master storage disk
subsystem that is also the source disk subsystem, and it just failed, and cannot communicate
any longer with its partners at the remote site.
The goal is to swap to the remote site and restart the applications. This requires, first, to make
the set of consistent volumes at the remote site available for the application, before the
application can be restarted at the remote site.
Then, once the local site is back and operational again, we must return to the local site.
Before returning to the local site, we must apply to the source volumes the changes that the
application did to the target data while it was running at the remote site. After this, we should
do a quick swap back to the local site and restart the application.

Server
FlashCopy
Global Copy A
A A
Primary
Primary
A
Primary
Primary
Primary
Primary
B
Primary
A
A
Primary target
A
Primary
source Primary
Primary
copy pending
copy pending C
Primary
tertiary

Figure 21-8 Production site fails
When the local storage disk subsystem fails, Global Mirror can no longer form Consistency
Groups. Depending on the state of the local storage disk subsystem, you may be able to
terminate the Global Mirror session. Usually this is not possible because the storage disk
subsystem may not respond any longer. Host application I/O may have failed and the
application ended. This goes usually along with messages or SNMP alerts that indicate the
problem. With automation in place, for example, eRCMF, these alert messages trigger the
initial recovery actions.
If the formation of a Consistency Group was in progress then, most probably, not all
FlashCopy relationships between the B an C volumes at the remote site will have reached the
corresponding point-in-time. Some FlashCopy pairs may have completed the FlashCopy
phase to form a new Consistency Group, and committed the changes already. Others may not
have completed yet, and are in the middle of forming their consistent copy, and remain in a
revertible state. And there is no master any longer to control and coordinate what may still be
going on. All of this imposes a closer look at the volumes at the remote site before we can
continue to work with them. There is more discussion of this in the following sections. First,
however, we bring the B volumes into a usable state using the failover command.
21.5.3 Global Copy Failover from B to A
Server for
GM recovery
Failover B to A
FlashCopy
Global Copy A
A A
Primary
Primary
A
Primary
Primary
Primary
Primary
B
Primary
A
A
Primary
source
source
A
Primary
Primary
Primary
copy pending
suspended C
Primary
tertiary

Figure 21-9 Perform Global Copy Failover from B to A
Because the source storage disk subsystem may no longer be usable, now the recovery
actions and processing occur at the remote site, using a server connected to the remote
storage disk subsystems for storage management console functions. See Figure 21-9 on
page 276.
You may use DS Storage Manager (DS SM), DS Command-Line Interface (DS CLI), and
eRCFM to execute the needed commands to the remote storage disk subsystems.
Doing a failover (Copy Services Failover function) on the Global Copy target volumes will turn
them into source volumes and suspend them immediately. This sets the stage for change
recording when application updates start changing the B volumes. Change recording in turn
allows you to re-synchronize the changes later from the B to the A volumes, before returning
to the local site to resume the application again at the local site. But at this stage the B
volumes do not contain consistent data — they are still useless. We just changed their state
from target copy pending to suspended. The A volumes state remains unchanged.
The key element when you run a Copy Services Failover is that the B volumes become the
new source volumes. This action just changes the state of the B volumes from target copy
pending to suspended. This action does not require communication with the other storage disk
subsystem at all, even though it is specified in the failoverpprc command. Once all the
failover commands are successfully executed, we can move on to the next step.
21.5.4 Verify for valid Consistency Group state

Now you must investigate whether all FlashCopy relationships are in a consistent state; see
Figure 21-10. This means that you must query all FlashCopy relationships between B and C,
which are part of the Consistency Group, to determine the state of the FlashCopy
relationship. Global Mirror might have been in the middle of forming a Consistency Group and
FlashCopy may have not completed the creation of a complete set of consistent C volumes.
Server for
GM recovery
Check Consistency Group
FlashCopy
Global Copy A
A A
Primary
Primary
A
Primary
Primary
Primary
Primary
B
Primary
A
A
Primary
source
source
A
Primary
Primary
Primary
copy pending
suspended C
Primary
tertiary

Figure 21-10 Check Consistency Group state
Each FlashCopy pair needs a FlashCopy query to identify its state. If the local site is still
accessible and also the source storage disk subsystem is accessible, you may consider
carrying these activities at the production site using remote FlashCopy commands. Most
likely the source storage disk subsystem does not respond any longer. In this case you must
target the query directly to the recovery site storage disk subsystem.

When you query a FlashCopy pair, there are two pieces of information that are key to
determine whether the C volume set is consistent or needs intervention: the revertible state
and the sequence number.
The lsflash command reports the Revertible state as either Enable or Disable, which
indicates whether the state of the FlashCopy is revertible or non-revertible. A non-revertible
state means that a FlashCopy process has completed successfully and all changes are
committed. Global Mirror uses the two-phase FlashCopy establishment operation. This
operation allows the storage disk subsystem to prepare for a new FlashCopy relationship
without altering the existing FlashCopy relationship. You can either commit or revert this new
FlashCopy relationship with revertible state by using the revertflash and commitflash
commands. During the Consistency Group formation process, Global Mirror puts all
FlashCopy relationships in the revertible state, and after they are in the revertible state,
commits all FlashCopy relationships. With this operation, the situation in which some
FlashCopy operatons have not started while others have completed does not occur.
The Sequence number is an identifier that can be set at FlashCopy establish operations and it
is then associated with the FlashCopy relationship. Subsequent FlashCopy withdraw
operations can be directed to FlashCopy relationships with specific sequence numbers.
Global Mirror uses the sequence number to identify a particular Consistency Group. The
actual sequence number used by Global Mirror is the platform timer from the Global Mirror
master storage disk subsystem (in seconds resolution) at the point when the Global Mirror
source components have to be coordinated to form a Consistency Group. This is at a point
before the Consistency Group is transferred to the remote site. If your master storage disk
subsystem platform timer is set to the time of day, then the FlashCopy sequence for Global
Mirror approximates a time stamp for the Consistency Group.
The best situation is when all the FlashCopy pairs of a Global Mirror session are in the
non-revertible state and all their sequence numbers are equal. No further action is necessary,
and the set of C volumes is consistent, and the copy is good.
Figure 21-11 shows the consistency group creation process. The action required depends on
the state of the consistency group creation process when the failure occurred.
Create consistency group by holding

application writes while creating Transmit updates in Global Copy mode
FlashCopy issued
bitmap containing updates for this while between consistency groups
with revertible option
consistency group on all volumes - Consistency group interval – 0s to 18hrs
design point is 2-3ms
Maximum coordination time (eg. 10ms) All FlashCopies
revertible
Drain consistency group and send FlashCopy committed Start next consistency group
to remote DS using Global Copy. once all revertible
Application writes for next Flashcopies have
consistency group are recorded in successfully completed
change recording bitmap Action required
Maximum drain time – eg.1 min
Figure 21-11 FlashCopy consistency group creation process
Depending on when the failure occurs, there are some combinations of revertible states and
FlashCopy sequence numbers that need different corrective actions. Use Table 21-1 as a
guide. This is a decision table and reads in the following way: When column 2 and column 3
are true, then take the action in column 4. Column 5 contains additional comments. Do this for
each of the four cases. The cases are described in chronological order, starting from the left.
Table 21-1 Consistency Group and FlashCopy validation decision table

Are all FC Are all FC sequence Action to take Comments
relationships numbers equal?
revertible?
Case 1 NO. YES. No action needed. All CG formation ended.

C volumes are
consistent.
Case 2 SOME - Some Revertible FlashCopy Revert FC relations. Some FlashCopy

FlashCopy pairs are pairs’ sequence pairs are running in a
revertible and others numbers are equal. Consistency Group
are not revertible. And non-revertible process and some
FlashCopy pairs have not yet started
sequence numbers their incremental
are equal, but do not process.
match the revertible
FlashCopies
sequence number.
Case 3 YES. YES. Revert to all FC All FlashCopy pairs

relations. are in a new
Consistency Group
process and none
have finished their
incremental process.
Case 4 SOME - Some YES. Commit FC relations. Some FlashCopy

FlashCopy pairs are pairs are running in a
revertible and others Consistency Group
are not revertible. process and some
have already finished
their incremental
process.
If you see a situation other than the above four situations, then the Global Mirror mechanism
has been corrupted.
Case 1: FlashCopies still committed

This indicates the situation where all FlashCopy operations have completed their task (and
the next FlashCopy operations have not been started). In this situation, we don’t need any
action to correct the FlashCopy status because we have consistent data in all C volumes.
This state is also reached after the FlashCopies are complete, and Global Mirror is waiting to
create the next Consistency Group.
Case 2: FlashCopies partially issued

In this case, there is a group of FlashCopy pairs that are all revertible and another group of
FlashCopy pairs that are all non-revertible. Consistency can be restored if these two criteria
are true:
򐂰 The FlashCopy sequence number for all revertible pairs is equal.
򐂰 The FlashCopy sequence number for all non-revertible pairs is equal too.

This indicates that the FlashCopy operations were interrupted. Some FlashCopy operations
for the new consistency group were started, but not all of them.
The FlashCopy relationships that have not started are in a non-revertible state and all of them
have the same FlashCopy sequence number. Other FlashCopy relationships, which had
already started, are in a revertible state and all of them have the same FlashCopy sequence
number, but the number is different from the sequence number of the non-revertible
FlashCopy relationships.
All these indications suggest that you have to return the revertible FlashCopy relationships to
the previous Consistency Group using the revertflash command, and terminate the
FlashCopy relationships.
Case 3: FlashCopies all revertible

In the case where all of the pairs are revertible and all FlashCopy sequence numbers are
equal, this indicates that all FlashCopy operations were running and none completed their
task for the Consistency Group formation. The fact that all relationships are still in a revertible
state denotes that nothing was finished and committed. Also, the identical FlashCopy
sequence numbers denote that all the FlashCopy operations were involved in the very same
Consistency Group set. All these indications suggest that you have to use the revertflash
command to return to the previous Consistency Group.
The revert action, which is invoked by the revertflash command, restores the Consistency
Group level between the source and target volumes to the prior state before the current
FlashCopy ran and resets the revertible state to Disable. The FlashCopy relationship is
preserved.
When the FlashCopy relationship is in a non-revertible state, the revert operation is not
possible. When you issue this command to FlashCopy pairs that are non-revertible, you are
going to see only an error message, but no action is performed.
Case 4: FlashCopies partially committed

If at the failure point some of the FlashCopy operations had completed their task to create a
consistent copy and committed this process, these operations will be non-revertible. Other
FlashCopy relationships may have not completed their corresponding part of the new
Consistency Group, so these will still be in a revertible state. If all of them, revertible and
non-revertible, have the same FlashCopy sequence number, this means that they all were
involved in the very same Consistency Group. This allows you to commit the revertible
FlashCopy relationships using the commitflash command.
To make the task easier, you may run the commitflash to all FlashCopy pairs. Non-revertible
FlashCopy relationships will just return an error message.
Usually all FlashCopy pairs are non-revertible and all sequence numbers are equal. In this
case, you do not need to take any action. Nevertheless, and depending on the failure
point-in-time, you may have to perform one of the above recovery actions. After this action, all
FlashCopy pairs will be non-revertible and all sequence numbers will be equal. Now you can
proceed to the next step.
21.5.5 Set consistent data on B volumes

At this point only the C volumes comprise a set of consistent data volumes. The B volumes
per definition do not provide consistent data, because Global Copy does not provide data
consistency. We want to have to two good copies of the data at the recovery site. The aim is
to have a consistent set of volumes to work with, still keeping a good copy to which we can
resort to if needed.
The next step then is to create the same consistency on the B volumes as we have on the C
volumes; see Figure 21-12. This can be achieved with the reverseflash command, with the
parameter -fast. This operation is called Fast Reverse Restore (FRR). You have to add the
-tgtpprc parameter with the reverseflash -fast command because the B volume is also the
Global Copy source at this step.
Server for
GM recovery
FRR FlashCopy
A
A A
Primary
Primary
A
Primary B
Primary
Primary
Primary
Primary
A
A
Primary GC source
FC source A
Primary
Primary
source C
Primary
Primary
suspended tertiary
pending
FC target

Figure 21-12 Set a consistent set of B volumes using the C volumes as source
Though the Fast Reverse Restore operation starts the background copy from C to B volumes,
in the reverseflash command you have to specify the B volumes as the FlashCopy sources
and the C volumes as the FlashCopy targets. With the reverseflash command you have to
use the following parameters:
򐂰 -fast
With this parameter, the reverseflash command can be issued before the background
copy completes. This option is intended for use as part of Global Mirror.
򐂰 -tgtpprc
After the Failover B to A operation described in 21.5.3, “Global Copy Failover from B to A”
on page 276, the B volume became a Global Copy source volume in suspended state.
The -tgtpprc parameter allows the FlashCopy target volume to be a Global Copy source
volume. You have to specify this parameter here because the B volume becomes a
FlashCopy target in the reverseflash process.
Because you do not specify the -persist parameter, the FlashCopy relationship ends after the
background copy from C to B completes.
The above Fast Reverse Restore (FRR) operation does a background copy of all tracks that
changed on B since the last Consistency Group (CG) formation. This results in the B volume
becoming equal to the image that was present on the C volume. This is the logical view. From
the physical data placement point of view, the C volume does not have meaningful data after
the FlashCopy relationship ends.
You have to wait until all Fast Reverse Restore operations complete successfully and its
background copy completes before you proceed with the next step. Again, when the
background copy completes, the FlashCopy relation will end. Therefore, you should check if
the FlashCopy relationships remain to determine when all Fast Reverse Restore operations
are completed.

21.5.6 Re-establish FlashCopy relationships between B and C
In this step you establish the former FlashCopy relationship between the B and C volumes, as
it was at the beginning when you set up the Global Mirror environment. This step is in
preparation for returning later to production at the local site. The command at this step is
exactly the same FlashCopy command you may have used when you initially created the
Global Mirror environment. See 20.3.4, “Introduce FlashCopy” on page 256.
In a disaster situation it may be that you do not want to use the -nocp option for the FlashCopy
from B to C. This will remove the FlashCopy I/O overhead when the application starts.
Now you may restart the applications at the remote site using the B volumes. Note the B
volumes are Global Copy source volumes in suspended state, which implies that change
recording takes place. Later this allows you to re-synchronize from B to A, before returning to
the local site.
Eventually, here you may also decide to create another copy of this consistent set of volumes,
and create D volumes, or preserve this set of consistent volumes on tape.
When you create the D volumes, this is just a plain FlashCopy command. Note that this is a
full volume FlashCopy. This is because at a previous step we re-established the FlashCopy
relationship between B and C, indicating it was part of a Global Mirror environment. This
indication carries the start change recording attribute, which implies that you can only use the
same source volume for full volume FlashCopy. Only a single incremental relationship is
allowed per source volume.
21.5.7 Restart the application at the remote site

At this stage you may restart the application at the remote site and work with the consistent
set of B volumes; see Figure 21-13.
Application
Server
Restart applications
a
Global Copy A
A A
Primary
Primary
Primary
A
Primary B
Primary
A
Primary
Primary
A
Primary source A
Primary
Primary
Primary
source suspended C
Primary
tertiary
pending

Figure 21-13 Restart applications at remote site
Note the suspended state of the B volumes, which implies change recording and indicates the
changed tracks on the B volumes. When the local site is about to become ready to restart the
applications again and resume the operations, you prepare the remote site for the next step.
21.5.8 Prepare to switch back to the local site
The local site is back and operative again. We assume that the local site did not lose the data
at the time when the swap to the remote site was done. It is then possible to re-synchronize
the changed data from B to A before re-starting the application at the local site. This is
accomplished by doing a Failback operation (Copy Services Failback function) from B to A;
see Figure 21-14.
Application
Server
Failback B to A
re-synchronize from B to A
A
A A
Primary
Primary
A
Primary
Primary
Primary
Primary
B
Primary
A
A
Primary
source A
Primary
target Global Copy
copy pending
Primary
Primary
C
Primary a
copy pending
tertiary

Figure 21-14 Failback operation from B to A in preparation for returning to local site
Note that the Failback operation is issued to the B volumes as the source and the A volumes
as the target. This command changes the A volume from its previous source copy pending
state to target copy pending and starts the re-synchronization of the changes from B to A.
Note: Before doing the Failback operation, ensure that paths are defined from the remote
site LSS to its corresponding LSS at the local site.
Note that with Fibre Channel links you can define paths in either direction on the very same
FCP link. During the Failback operation the application stays running at the remote site to
minimize the application outage.
If the A volume is still online to the server at the local site or it was online when a crash
happened, so that the SCSI persistent reserve is still set on the previous source disk (A
volume), the Global Copy Failback process with the failbackpprc command fails. In this case
the server at the production site locks the target with a SCSI persistent reserve. After this
SCSI persistent reserve is reset with the varyoffvg command (in this case on AIX), the
failbackpprc command completes successfully.
There is a -resetreserve parameter for the failbackpprc command. This option resets the
reserved state so that the failback operation can complete. In the Failback operation after a
real disaster, you may use this parameter because the server might go down while the SCSI
persistent reserve was set on the A volume. In the planned failback operation, you must not
use this parameter because the server at the local site still owns the A volume and may be
using it and the Failback operation suddenly changes the contents of the volume. This may
cause the server file system’s corruption.

21.5.9 Return to the local site
Once the local site is ready, quiesce the application at the remote site. Then a sequence of
Global Copy Failover - Failback operations from A to B will re-establish Global Copy back as it
was originally before the local site outage.
Server for Application

GM recovery Server
Failover A to B
A
A A
Primary
Primary
A
Primary
Primary
Primary
Primary
B
Primary
A
A
Primary
source Global Copy
source A
Primary
Primary
Primary
copy pending C
Primary
suspended
tertiary

Figure 21-15 Global Copy Failover from A to B
Figure 21-15 shows the action at the local site, the Global Copy Failover operation from A to
B. This failoverpprc command will change the state of the A volumes from target copy
pending to source suspended, and start to keep a bitmap record of the changes to the A
volumes. You issue this command to the storage disk subsystem at the local site. The state of
the B volumes does not change.
Once the failover is completed, a failback operation from A to B is run; see Figure 21-16.
Server for Server

GM recovery
Failback A to B
Re-synchronize from A to B
A
A A
Primary
Primary
A
Primary
Primary
Primary
Primary
B
Primary
A
A
Primary
target
A
Primary
source Global Copy Primary
Primary
copy pending
copy pending C
Primary
tertiary

Figure 21-16 Global Copy failback from A to B and resync Global Copy volumes
Note: Before doing the Failback operation, ensure that paths are defined from the local site
LSS to its corresponding LSS at the remote site.
Figure 21-16 on page 284 shows the Failback operation at the local site. The failbackpprc
command will change the state of the A volumes from source suspended to source copy
pending. The state of the B volumes will change from source copy pending to target copy
pending. Also, the replication of updates from A to B begins. This replication ends quickly
because the application did not start yet at the local site.
Finally, if you did not already establish the FlashCopy relationships from B to C during the
Failover-Failback sequence at the remote site, then you have to do it now. This may be an
inband FlashCopy, as shown in Figure 21-17.
Server for
GM recovery
Remote FlashCopy B to C
FlashCopy
A
A A
Primary
Primary
A
Primary
Primary
Primary
Primary
B
Primary
A
A
Primary
source Global Copy
target
A
Primary
Primary
Primary
copy pending
copy pending C
Primary
tertiary
FlashCopy target
Figure 21-17 Establish Global Mirror FlashCopy relationship between B and C
The last step is to start the Global Mirror session again, as shown in Figure 21-18. Then the
application may resume at the local site.
Application Server for

server GM recovery
I/O Start GM session
FlashCopy
A
A A
Primary
Primary
A
Primary
Primary
Primary
Primary
B
Primary
A
A
Primary
source Global Copy
target
A
Primary
Primary
Primary
copy pending
copy pending C
Primary
Tertiary
FlashCopy target
Figure 21-18 Start Global Mirror session and resume application I/O at local site

21.5.10 Conclusions of failover/failback example
This concludes the sequence of steps for a swap to the remote site and coming back to the
local site after service is restored at the local site.
In particular, the check on a valid Consistency Group after a production site failure is a
challenge when you consider a large configuration with many volume pairs. Each command
usually addresses a single pair of volumes. When you have a large quantity of volume pairs,
this requires automation, for example, to check all the FlashCopy relationships after a
production site failure. Note that for a planned site swap, you may not have to check for a valid
Consistency Group because, through a proper command sequence and verifying their
successful completion, the possibility of an inconsistent group of C volumes is very minimal, if
at all.
Global Mirror is a 2-site solution that can bridge any distance between both sites. There are
ready-to-use packages and services available to implement a disaster recovery solution for
2-site remote copy configurations. There is eRCMF that supports not only Global Copy and
Metro Mirror based configurations, but also Global Mirror configurations.
22
Chapter 22. Global Mirror interfaces

In this chapter we provide an overview of the interfaces you can use to manage and control
Global Mirror environments.
The information discussed in this chapter can be complemented with the following sources:
򐂰 Part 2, “Interfaces” on page 15
򐂰 Chapter 8, “FlashCopy interfaces” on page 63
򐂰 Chapter 17, “Global Copy interfaces” on page 201
򐂰 The examples presented in Chapter 24, “Global Mirror examples” on page 305
򐂰 The publication IBM System Storage DS8000 Command-Line Interface User´s Guide,
SC26-7916

22.1 Overview
Global Mirror combines Global Copy and FlashCopy, which work together in an autonomic
fashion under the DS8000 microcode control. There are commands intended for Global Copy
and FlashCopy, as well as commands that address Global Mirror sessions. All of them can be
managed with the following interfaces:
򐂰 DS Storage Manager (DS SM) graphical user interface (GUI)
򐂰 enterprise Remote Copy Facility (eRCMF)
This chapter gives an overview of the DS CLI and the DS SM for Global Mirror management.

eRCMF is a scalable and flexible solution to protect business data and maintain consistent
data, providing an automation tool that simplifies the disaster recovery operations. For more
information about this solution see Part 9, “Solutions” on page 537.
DS CLI and DS SM similar functions for Global Mirror management

Table 22-1 lists DS CLI commands and equivalent DS SM panel options for Global Mirror
management.
Table 22-1 DS CLI commands and equivalent DS SM panel options

Task DS CLI DS SM
Start Global Mirror mkgmir CS → GM → Create

session
Stop Global Mirror rmgmir CS → GM → Delete

session
Pause Global Mirror pausegmir CS → GM → Pause

session
Resume Global Mirror resumegmir CS → GM → Resume

session
Show Global Mirror showgmir CS → GM →

status Properties
Create a Global Mirror mksession CS → GM → Create

session on an LSS Or
CS → GM → Modify
Remove a Global rmsession CS → GM → Delete

Mirror session from an Or
LSS CS → GM → Modify
Change a Global chsession CS → GM → Modify

Mirror session on an
LSS
Task DS CLI DS SM
Display a Global Mirror lssession CS → GM → View

session on an LSS session volumes
Create a complete mkpprcpath CS → P → Create

Global Mirror mkpprc CS → MM → Create
environment mkflash CS → FC → Create
mksession CS → GM → Create
mkgmir
Fail over Global Mirror rmgmir CS → GM → Delete

failoverpprc CS → MM → Failover
lsflash CS → FC →
revertflash Properties
or CS → FC → Reverse
commitflash relationship
reverseflash CS → FC → Create
mkflash
Fail back Global Mirror mkpprcpath CS → P → Create

failbackpprc CS → MM → Failback
lspprc CS → MM →
mkpprcpath Properties
failbackpprc CS → MM → Failback
resumegmir CS → GM → Resume
Practice Failover pausegmir CS → GM → Pause

Global Mirror showgmir CS → GM →
pausepprc Properties
lsflash CS → FC →
reverseflash Properties
mkflash CS → FC → Reverse
relationship
CS → FC → Create
Practice Failback failbackpprc CS → MM → Failback

Global Mirror resumegmir CS → GM → Resume
Clean up a complete rmgmir CS → GM → Delete

Global Mirror chsession CS → FC → Delete
environment rmsession CS → MM → Delete
rmflash CS → P → Delete
rmpprc
rmpprcpath
CS: Stands for Copy Services menu.

GM: Stands for Global Mirror panel.
MM: Stands for Metro Mirror panel.
P: Stands for Paths panel.
FC: Stands for FlashCopy panel.
22.2 DS Command-Line Interface

The DS Command-Line Interface can be used to set up and manage Global Mirror functions.
One big usability advantage of the DS CLI is that all platforms use a common syntax. This
can make it easier to learn, and make it a common tool for skilled people as well as lesser
Chapter 22. Global Mirror interfaces 289

skilled people. Another big usability advantage is that you can make scripts with the
commands, which saves lots of time when you have to execute several predetermined tasks.
The DS CLI commands are issued via the Ethernet network to the DS Storage Management
Console (DS HMC). The DS CLI server resides on the DS HMC, which works as a gateway
between the clients and the storage disk subsystem. You can install a DS CLI client to access
the DS CLI server on various operating systems. See 14.2, “Copy Services network
components” on page 135, for a detailed description of the DS HMC network environment.
The DS CLI commands that are used for managing a Global Mirror environment fall into the
following basic groups:
򐂰 For paths definitions and removal
򐂰 For Global Copy pairs creation, management, and removal
򐂰 For FlashCopy pairs creation, management, and removal
򐂰 For Global Mirror session definition, management, and removal
The first three groups in the previous list refer to the same commands you use to manage and
control Global Copy and FlashCopy environments. Still, when working in a Global Mirror
environment specific command parameters must be used for some of the tasks. How these
commands are used and the options that must be selected is covered in the discussions and
examples we present in this part of the book.
For the Global Mirror session itself and its definition, management, modification, and removal,
we have the following DS CLI commands:
򐂰 showgmir displays detailed properties and performance figures for Global Mirror.
򐂰 showgmiroos displays the number of out of synchronization (out of sync) tracks for the
session.
򐂰 mkgmir starts Global Mirror.
򐂰 rmgmir stops Global Mirror.
򐂰 pausegmir pauses Global Mirror.
򐂰 resumegmir resumes Global Mirror.
򐂰 mksession opens a Global Mirror session on a LSS.
򐂰 chsession allows you to modify a Global Mirror session on a LSS. You can add and
remove a volume for Global Mirror with this command.
򐂰 rmsession removes an existing Global Mirror session on a LSS.
򐂰 lssession displays a Global Mirror session on a LSS.
For most DS CLI commands, you will need to know some (or all) of the following information:
򐂰 The serial number and device type of the source and target storage disk subsystem.
򐂰 The World Wide Node name (WWNN) of the remote storage disk subsystem.
򐂰 The LSS numbers of the source and target volumes.
򐂰 The port IDs for source and target. Up to eight port pair IDs can be specified.
A full establishment of a Global Mirror environment using DS CLI commands may take a long
time, specially if you need to set up an environment involving many volumes on many LSSs
and in several storage disk subsystems. For the setup and management of large-scale
environments, eRCMF or TPC for Replication are more appropriate, especially if you want to
test disaster recovery scenarios or if you want to run a real disaster recovery plan.
Detailed examples of how to set up and manage a Global Mirror environment using DS CLI
commands can be found in Chapter 24, “Global Mirror examples” on page 305.
22.3 DS Storage Manager GUI

You can use the DS Storage Manager (DS SM) graphical user interface (GUI) to set up and
manage some Global Mirror functions. You work with the DS SM GUI through a series of
predetermined panels and options you can select to accomplish the desired task. The DS SM
operates from a Web session that communicates to the DS Hardware Management Console
(DS HMC). See 14.2, “Copy Services network components” on page 135, for a detailed
description of the DS HMC network environment.
The DS SM interface is user friendly. However, you cannot use it for automation activities, and
certain Copy Services functions are not supported from this interface.
People who have fewer skills or less confidence with the other interfaces can use the DS
Storage Manager interface. It is reasonably simple to use; however, it is usually slower than
the other interfaces, and you cannot save actions for later use.
For Global Mirror setup activities, you will start from the following DS Storage Manager
panels:
򐂰 For paths definitions, start from the Paths panel under the Copy Services menu. You can
see this panel in Figure 24-16 on page 356.
򐂰 To create Global Copy pairs, start from the Metro Mirror panel under the Copy Services
menu. You can see this panel in Figure 24-23 on page 359.
򐂰 To create FlashCopy pairs, start from the FlashCopy panel under the Copy Services
menu. You can see this panel in Figure 24-30 on page 364.
򐂰 For Global Mirror session setup and control, start from the Global Mirror panel under the
Copy Services menu of the DS SM; see Table 22-1 on page 288.
Figure 22-1 Global Mirror main panel

If you do not select any check box for a Global Mirror session, the pull-down list only allows
session creation. The pull-down list will look like Figure 22-2. If you are working with a storage
disk subsystem that has more than one storage image, then in this panel you can also use the
storage complex, storage unit, and storage image pull-down lists to access other storage
images with which the Global Mirror session might be operating.
Figure 22-2 Global Mirror pull-down list - no session selected
If you select any check box for a Global Mirror session, the pull-down list will allow session
creation and management. The pull-down list will look like Figure 22-3.
Figure 22-3 Global Mirror pull-down list - Session selected
Consider that a full establishment of a Global Mirror environment requires using each Copy
Services panel in the DS SM and, if you need to set up an environment involving many
volumes on many LSSs and in several storage disk subsystems, this can take a very long
time. Therefore, the DS SM is more convenient for specific non-large-scale Copy Services
activities, and also it is a very didactic tool for users not skilled in other interfaces.
For the setup and management of large-scale environments, eRCMF or TPC for Replication
are more appropriate, especially if you want to test disaster recovery scenarios or if you want
to run a real disaster recovery plan.
Detailed examples of how to set up and manage a Global Mirror environment using the DS
Storage Manager GUI can be found in Chapter 24, “Global Mirror examples” on page 305.
22.4 eRCMF
The enterprise Remote Copy Management Facility (eRCMF) can be used to manage Metro
Mirror, Global Copy, and Global Mirror functions, as well as their associated FlashCopy tasks.
eRCMF can be use to manage Copy Services functions in a mixed open systems and z/OS
environment. It is a powerful tool that can consolidate the equivalent of several DS CLI
commands with just one command. Nevertheless, it is not using DS CLI scripts.
eRCMF is convenient for the management of medium and large scale environments. You will
find eRCMF indispensable if you want to test disaster recovery scenarios or if you want to run
a real disaster recovery plan.
eRCMF allows the automation of the recovery activities. It will not only trap SNMP messages,
but it will also execute commands in order to shorten downtime.
For a detailed description of eRCMF, from its installation through its configuration to its usage,
see Chapter 36, “IBM TotalStorage Rapid Data Recovery for UNIX and Windows” on
page 575.

23
Chapter 23. Global Mirror performance and

scalability
This chapter discusses performance considerations for planning and configuring Global
Mirror for DS8000. Also discussed in this chapter is the potential impact that the three phases
of Consistency Group formation might have on application write I/Os. Also covered is the
distribution of the B volumes and the C volumes across different ranks, and providing extra
care for very busy volumes.

23.1 Performance aspects for Global Mirror
Global Mirror is basically comprised of Global Copy and FlashCopy and combines both
functions to create a solution that provides consistent data at a distant site. This means that to
have a satisfactory Recovery Point Objective (RPO) without impacting production too much,
performance should be analyzed at the production site, at the recovery site, and between
both sites.
򐂰 At the production site, even if production is a higher priority than replication, the storage
disk subsystem has to handle both production load and Global Copy replication load. If the
source storage disk subsystem is overloaded by production, this will slow the Consistency
Group formation process because it needs to wait for the drain of the out-of-sync tracks.
򐂰 Between both sites, the bandwidth needs to be sized for production load peaks.
򐂰 At the recovery site, even if production is not running, it is hosting the target Global Copy
volumes and also handling the FlashCopy operations. Therefore performance of the
remote storage disk subsystem also needs some analysis to ensure the best possible
RPO.
This section discusses the impact of Global Copy and FlashCopy in the overall performance
of Global Mirror. Global Copy has at most only a minimal impact to the response time of an
application write I/O to a Global Copy source volume.
In the Global Mirror environment FlashCopy is used with the nocopy attribute. This implies
that, for any write I/Os to the FlashCopy source volume, there will be additional internally
triggered I/Os in the remote storage disk subsystem. These I/Os preserve the FlashCopy
source volume tracks by making a copy to the FlashCopy target volume before the source
tracks are updated. This happens every time within the interval in between two FlashCopy
establish operations.
Note that FlashCopy with the nocopy option does not start a background copy operation, but
it only maintains a set of FlashCopy bitmaps for the B and C volumes. These bitmaps are
established the first time a FlashCopy relationship is created with the nocopy option. Before a
source track is modified between the creation of two Consistency Groups, the track is copied
to the target C volume to preserve the previous point-in-time copy. This includes updates to
the corresponding bitmaps to reflect the new location of the track, which belongs to the
point-in-time copy. Each Global Copy write to its target B volume within the window of two
adjacent Consistency Groups causes such a FlashCopy I/O operation. See Figure 23-1.
1 Write A 2 3 Write A Read Write Primary

Primary
Read Primary
A1
Primary
Primary B1
Primary C1
source target tertiary
4 5
copy pending FCP links copy pending
Host Local site Remote site
Figure 23-1 Global Copy with write hit at the remote site
An I/O in this context is also a Global Copy I/O when Global Copy replicates a track from a
Global Copy source volume to a Global Copy target volume. This implies an additional read
I/O in the source storage disk subsystem at the local site and a corresponding write I/O to the
target storage disk subsystem at the remote site. In a Global Mirror environment, the Global
Copy target volume is also the FlashCopy source volume. This may have some effect on a
very busy disk subsystem.
Figure 23-1 on page 296 roughly summarizes what happens between two Consistency Group
creation points when application writes come in. The application write I/O completes
immediately at the local site (1). Eventually Global Copy replicates the application I/O and
reads the data at the local site (2). Because of the FlashCopy nocopy attribute, before the
track gets updated on the B volume it is first preserved on the C volume.
Note: What Figure 23-1 and Figure 23-2 show is not necessarily the exact sequence of
internal I/O events, but a logical view approximation. There are internal microcode
optimization and consolidation techniques that make the entire process more efficient.
What is shown in Figure 23-1 on page 296 is the normal sequence of I/Os within a Global
Mirror configuration. The write (3) to the B1 volume is usually a write hit in the persistent
memory (NVS) of the target disk subsystem, so this is an instant operation. Eventually at
some later moment the B1 track is copied from B1 to C1 so it can be preserved and the
update in NVS can be destaged to the B1 volume.
There is some potential impact on the Global Copy data replication operation depending on
whether persistent memory or non-volatile cache is overcommitted in the target storage disk
subsystem. This may cause the FlashCopy source tracks to have to first be preserved on the
target volume C1 before the Global Copy write is completed. See Figure 23-2.
1 Write A 2 5 Write A Read Write Primary

Primary
Read Primary
A1
Primary
Primary B1
Primary C1
source target tertiary
3 4
copy pending FCP links copy pending
Host Local site Remote site
Figure 23-2 Global Copy with overcommitted NVS at the remote site
Figure 23-2 roughly summarizes what happens when persistent memory or NVS in the
remote storage disk subsystem is overcommitted. A read (3) and a write (4) to preserve the
FlashCopy source track and write it to the C1 volume are required before the write (5) can
complete. Then, when the track is updated on the B1 volume, this completes the write (5)
operation. Nevertheless, what you should usually see is all quick writes to cache and
persistent memory as Figure 23-1 on page 296 outlines.
All write I/Os to FlashCopy source volumes also trigger bitmap updates. This is the bitmap
that was created when the FlashCopy volume pair was created with the start change
recording attribute. This allows the replication of only the changed recording bitmap to the
corresponding bitmap for the target volume in the course of forming a Consistency Group.
See 20.4, “Consistency Groups” on page 259.
Chapter 23. Global Mirror performance and scalability 297

23.2 Performance considerations at coordination time
When looking at the three phases that Global Mirror goes through to create a set of data
consistent volumes at the remote site, the first question that comes to mind is whether or not
the coordination window imposes an impact on the application write I/O; see Figure 23-3.
Maximum Maximum
coordination drain
time time
Serialize all Perform

1 Global Copy 2 Drain data from local to remote site 3 FlashCopy
source volumes
Write
Hold write I/O
Global Copy paths

A1 B1
source target C1
I/O Global Mirror tertiary
session paths
A2 B2
source Global Copy paths target C2
tertiary
Figure 23-3 Coordination time: How it impacts application write I/Os
The coordination time, which you can limit by specifying a number of milliseconds, is the
maximum impact to an application’s write I/Os that will be allowed when forming a
Consistency Group. The intention is to keep this time window as small as possible. The
default of 50 ms may be a bit high in a transaction processing environment. A valid number
may also be a very low number — in the single-digit range. The required communication
between the master storage disk subsystem and the subordinate disk subsystems is inband
over the paths between the master and the subordinates. This communication is highly
optimized and allows you to minimize the potential application write I/O impact to 3 ms, for
example. Note that this communication is performed over FCP links. There is at least one
FCP link required between a master storage disk subsystem and a subordinate. For
redundancy, we suggest using two FCP links.
The following example illustrates the impact of the coordination time when Consistency Group
formation starts, and whether this impact has the potential to be significant.
Assume a total aggregate number of 5000 write I/Os over two source disk subsystems with
2500 write I/Os per second to each disk subsystem. Each write I/O takes 0.5 ms. You
specified 3 ms maximum to coordinate between the master the subordinate disk subsystems.
Assume further that a Consistency Group is created every 3 seconds, which is a goal set with
the Consistency Group interval time of zero.
򐂰 5000 write I/Os.
򐂰 0.5 ms response time for each write I/O.
򐂰 Maximum coordination time is 3 ms.
򐂰 Every 3 seconds a Consistency Group is created.
This is 5 I/Os for every millisecond or 15 I/Os within 3 ms. So each of these 15 write I/Os
experience a 3 ms delay. This happens every 3 seconds. Then we observe an average
response time delay of approximately:
(15 IOs * 0.003 sec) / 3*5000 IO/sec) = 0.000003 sec or 0.003 ms
The response time increases in average from 0.5 ms to 0.503 ms.
23.3 Consistency Group drain time

Once the Consistency Group is set at the source disk subsystem by setting the corresponding
bitmaps within the coordination time window, all remaining data that is still in the out-of-sync
(OOS) bitmap is sent (drained) by Global Copy to the target disk subsystem.
This drain period may also be limited to a maximum drain time. The default is 30 seconds and
is considered to be too small in a potentially write-intensive workload. Consider a number in
the range of 300 seconds to 600 seconds.
This replication process usually does not impact the application write I/O. There is a very low
possibility for a track within a Consistency Group to be updated before this track is replicated
to the remote site within this drain time period. When this unlikely event happens, the track is
immediately replicated to the target disk subsystem before the application write I/O modifies
the original track. The involved application write I/O experiences a similar response time delay
as though the I/O had been written to a Metro Mirror source volume. Note that subsequent
writes to this same track do not experience any delay because the track has already been
replicated to the remote site.
23.4 Remote storage disk subsystem configuration

There will be I/O skews and hot spots in the storage disk subsystems. This is true for the local
and remote disk subsystems. In local disk subsystems you may consider a horizontal pooling
approach and spread each volume type across all ranks. Volume types in this context are, for
example, DB2 database volumes, logging volumes, batch volumes, and temporary volumes.
Your goal may be to have the same amount of each volume type within each rank.
Through a one-to-one mapping from a local to a remote disk subsystem you achieve the
same configuration at the remote site for the B volumes and the C volumes. Figure 23-4 on
page 300 proposes spreading the B and C volumes across different ranks.

The goal is to put the same numbers of each volume type into each rank. For volume type
here, we refer to B volumes and C volumes within a Global Mirror configuration. In order to
avoid performance bottlenecks, spread busy volumes over multiple ranks. Otherwise, hot
spots may concentrate on single ranks when you put the B and C related volumes on the very
same rank. We recommend spreading B and C volumes as Figure 23-4 suggests. Another
approach can be to focus on the very busy volumes and keep these volumes on separate
ranks from all of the other volumes.
Primary
A Primary
A Primary
B1
Primary C3
A1
Primary
Primary target tertiary
source
copy pending
copy pending
Rank 1 Rank 1
Primary
FCP links A Primary
A FCP FCP
Primary
B2
Primary C1
A2
Primary
Primary
target tertiary
source
copy pending
copy pending
Rank 2 Rank 2
Primary
A Primary
A Primary
B3
Primary C2
A3
Primary
source
copy pending
Host copy pending
Rank 3 Rank 3
Figure 23-4 Remote disk subsystem with all ranks containing equal numbers of volumes
With mixed Disk Drive Module (DDM) capacities and different speeds at the remote storage
disk subsystem, you may consider spreading B volumes not only over the fast DDMs but over
all ranks. Basically, follow a similar approach, as Figure 23-4 recommends. You may keep
particularly busy B volumes and C volumes on the faster DDMs.
Figure 23-5 shows a configuration that incorporates the D volumes, which you can create
once in a while for test or other purposes.
Primary
A Primary
A Primary
B1
Primary C3
A1
Primary
source
copy pending
copy pending
Rank 1 Rank 1
Primary
FCP links A Primary
A FCP FCP
Primary
B2
Primary C1
A2
Primary
Primary
target tertiary
source
copy pending
copy pending
Rank 2 Rank 2
Primary
A Primary
A Primary
B3
Primary C2
A3
Primary
source
copy pending
Host copy pending
Rank 3 Rank 3
Local site
Primary
A Primary
Primary
Host D1
Primary D2
Primary
A Primary
Remote Primary
D3
Primary D4
site
Rank 4
Figure 23-5 Remote disk subsystem with D volumes
For a situation like the one illustrated in Figure 23-5 we suggest as an alternative, a rank with
larger and perhaps slower DDMs. The D volumes may be read from another host, and any
other I/O to the D volumes does not impact the Global Mirror volumes in the other ranks.
Note that if you use the nocopy option for the relationship between the B and D volumes, this
will read the data from B when read I/Os to the D volume happen. So for this circumstance
you may consider using the copy option, thus preventing additional I/O to the ranks with the B
volumes. However, in this case, until the background copy between B and D completes, there
may be some impact to the Global Mirror data transfer.
An option here may be to spread all B volumes across all ranks again and also configure the
same number of volumes in each rank. Still, put the B and C volumes in different ranks. We
further recommend that you configure corresponding B and C volumes in such a way that
these volumes have an affinity to the same server. It would be ideal to also have the B
volumes connected to a different DA pair than the C volumes.
23.5 Balancing the disk subsystem configuration

In this section we show examples of how to gather information and do the analysis of how well
the storage disk subsystem is balanced in relation to the I/O load.
Example 23-1 on page 302 shows showgmiroos command outputs, which display
OutOfSyncTracks for the storage image scope as well as two LSS scopes. The first
showgmiroos command with the -scope si parameter shows that some Out Of Sync Tracks
remain within the disk subsystem. The second showgmiroos command with the -scope lss

parameter shows OutOfSyncTracks with a value of 0 (zero). This means that for LSS 10 there
are no tracks remaining at the local site that have not been transferred yet to the remote site.
This is different with LSS 11, where there is still data that has not yet been transferred to the
remote site. The third showgmiroos command shows that there are 67,847 tracks or roughly
4.14 GBs still waiting to get replicated from the local to the remote site. This situation denotes
a significant write I/O skew between LSS 10 and LSS 11.
Example 23-1 Out Of Sync Tracks shown by the showgmiroos command

dscli> showgmiroos -dev IBM.2107-7520781 -lss 10 -scope si 02
Scope IBM.2107-7520781
Session 02
OutOfSyncTracks 73125
dscli> showgmiroos -dev IBM.2107-7520781 -lss 10 -scope lss 02
Scope IBM.2107-7520781/10
Session 02
OutOfSyncTracks 0
dscli> showgmiroos -dev IBM.2107-7520781 -lss 11 -scope lss 02
Scope IBM.2107-7520781/11
Session 02
dscli>
Example 23-2 shows the lspprc command that reports the Out Of Sync Tracks by volume
level. Here we see that two volumes in LSS 10 have no Out Of Sync Tracks and two volumes
in LSS 11 have some number of Out Of Sync Tracks.
This particular configuration has source and target FlashCopy volumes within the very same
rank. Either you may consider putting one of the two busy volumes in LSS 11 into LSS 10 so
as to distribute the load over both LSSs, or another approach is to configure the FlashCopy
source volume and its corresponding target into different ranks or LSSs. See 23.4, “Remote
storage disk subsystem configuration” on page 299.
Example 23-2 Out Of Sync Tracks shown by the lspprc command

dscli> lspprc -l 1000-1001 1100-1101
====================================================================================================================
dscli>
When the load distribution is unknown in your configuration, you may consider developing
some rudimentary code based on a script, for example, which regularly issues the
showgmiroos commands (as shown in Example 23-1) and the lspprc -l command )as
shown in Example 23-2). You may then process the output of these commands to better
understand the write load distribution over the Global Copy source volumes.
Note that the numbers in Example 23-1 may be just a brief peak period. It is still feasible to
use the conventional approach with I/O performance reports, such as iostat in the UNIX
environment, to investigate the write workload. Or Tivoli Productivity Center for Disk could be
used to analyze the storage disk subsystem performance.
23.6 Growth within Global Mirror configurations
When you add a rather large number of volumes at once to an existing Global Mirror session,
then the available resources for Global Copy within the affected ranks might be over-utilized
or even monopolized by the initial copy pass. To avoid too much impact, consider adding
many new volumes to an existing Global Mirror session in stages. If possible, and as a rough
rule of thumb, add only a few volumes in a rank during application peak I/O periods. Once the
first initial copy is complete, add the next few volumes. Again, as a rule of thumb, plan for
adding one or two volumes in a rank during peak I/O load for the first initial copy pass. If
possible, then plan a massive add of new Global Copy volumes into an existing session
during off-peak periods.

24
Chapter 24. Global Mirror examples

This chapter provides examples that illustrate how to set up and manage an Global Mirror
environment using the DS CLI and the DS Storage Manager GUI. The examples describe
how to:
򐂰 Set up Global Mirror (paths, volume pairs, session).
򐂰 Manage Global Mirror.
򐂰 Remove the environment.
򐂰 Manage a site swap from the production to the recovery site and back.
The information discussed in this chapter is complemented with the publication IBM System
Storage DS8000 Command-Line Interface User´s Guide, SC26-7916.

24.1 Set up a Global Mirror environment using the DS CLI
In this section we present an example of how to set up a Global Mirror environment using the
DS CLI. Figure 24-1 shows the configuration we are going to implement.
In this configuration, different LSS and LUN numbers were used across the A, B, and C
components, so that you could unmistakably identify every element when referenced along
the discussion.
Note: In a real environment, and differently from our example, to simplify the management
of your Global Mirror environment, it is better for you to maintain a symmetrical
configuration in terms of both physical and logical elements.
.
FlashCopy
Global Copy
A B C
LSS10 LSS20 FlashCopy LSS22
Global Copy paths
GM Master
1000 FCP link 2000 2200
1001 2001 2201
1100 2100 2300

FCP link
1101 2101 2301
Global Copy paths
DS8000#1 (local) DS8000#2 (remote)
Figure 24-1 DS8000 configuration in the Global Mirror example

Before starting the tasks to configure the Global Mirror environment, we recommend that you
first do an initial DS CLI setup similar to the one explained in 14.5.1, “Preparing to work with
the DS CLI” on page 137. This initial DS CLI setup will allow a simpler syntax of the
commands you will be using to configure the Global Mirror environment.
24.1.2 Configuration used for the environment

Figure 24-1 shows the configuration used for this example. The configuration has four A
volumes residing in two LSSs on DS8000#1, four B volumes residing in two LSSs on
DS8000#2, and four C volumes residing in two other LSSs also on DS8000#2. Two paths are
defined for each Global Copy source and target LSS pair (LSS10:LSS20 and LSS11:LSS21).
We start the Global Mirror master in LSS10.
24.1.3 Setup procedure
The sequence of steps for the creation of a Global Mirror environment is not completely fixed,
and allows for some variation. Still, we recommend that you follow this procedure:
1. Create Global Copy relationships (A to B volumes).
2. Create FlashCopy relationships (B to C volumes).
3. Start Global Mirror session.
24.1.4 Create Global Copy relationships - A to B volumes

The first step of the procedure is to create the Global Copy relationships between the A and
the B volumes. For this, you must do the following:
1. Determine the available FCP links between the local and the remote disk subsystems.
2. Create the Global Copy paths between the local and the remote LSSs.
3. Create the Global Copy volume pairs.
4. Wait until the first copy of the Global Copy pairs completes.
This procedure is followed in Example 24-1 where you can see the sequence of commands
and the corresponding results.
Note that the tasks you must perform to create the Global Copy relationships in a Global
Mirror environment are similar to what has been presented in Part 5, “Global Copy” on
page 185. You can refer to 19.1.2, “Set up of Global Copy configuration” on page 215.
Example 24-1 Create Global Copy pairs relationships (A to B)

<< Determine the available fibre links >>
===================================================
dscli>
===================================================
dscli>
<< Create paths >>

-tgtlss 20 i0143:i0010 i0213:i0140
dscli>
-tgtlss 21 i0143:i0010 i0213:i0140
dscli>
=========================================================
10 20 Success FF20 I0143 I0010 5005076303FFC663
10 20 Success FF20 I0213 I0140 5005076303FFC663
11 21 Success FF21 I0143 I0010 5005076303FFC663
Chapter 24. Global Mirror examples 307

11 21 Success FF21 I0213 I0140 5005076303FFC663
dscli>
<< Create Global Copy pairs >>

dscli>
dscli> lspprc -l 1000-1001 1100-1101
====================================================================================================================

dscli>
<< wait to see that the Out Of Sync Tracks shows 0 >>
dscli> lspprc -l 1000-1001 1100-1101

=====================================================================================================================

dscli>
24.1.5 Create FlashCopy relationships - B to C volumes

To create the FlashCopy relationships between the B and C volumes, use the mkflash or the
mkremoteflash command with the following parameters: -tgtinhibit, -record, and -nocp. The
-persist parameter is automatically selected when the -record parameter is selected.
Therefore you do not specify the -persist parameter explicitly. Following is a brief explanation
for each parameter; see also 20.3.4, “Introduce FlashCopy” on page 256:
򐂰 -tgtinhibit
Prevents host system writes to the target while the FlashCopy relationship exists.
򐂰 -record
Keeps record OF the tracks that were modified on both volumes within a FlashCopy pair.
Select this parameter when you create an initial FlashCopy volume pair that you intend to
use with the resyncflash command.
򐂰 -nocp
Inhibits background copy. Data is copied from the source volume to the target volume only
if a track on the source volume is modified.
򐂰 -persist
Keep the FlashCopy relationship until explicitly or implicitly terminated.
Depending on your network environment, you can give the FlashCopy command to the local
DS8000#1 for its inband transmission to the remote DS8000#2. In this case you use the
mkremoteflash command. Alternatively, if you have connectivity to the remote DS8000#2,
then you can give the mkflash command directly to the DS8000#2.
In our example we use the inband functionality of FlashCopy, in which case we have to
specify the LSS having the A volume for the -conduit parameter and the storage image ID at
the remote site for the -dev parameter; see Example 24-2. You have to give this command to
the DS HMC connected to the local DS8000#1.
Because the -nocp parameter is specified and the Global Copy initial copy (first pass)
completed in the previous step, no FlashCopy background copy occurs at this time.
Note: You can create this FlashCopy relationship before the initial copy of Global Copy.
However, to avoid unnecessary FlashCopy background I/Os we do not recommend it.
Example 24-2 Create FlashCopy relationship - B to C volumes

dscli> mkremoteflash -tgtinhibit -nocp -record -conduit IBM.2107-7520781/10 -dev IBM.2107-75ABTV1 2000-2001:2200-2201
CMUC00173I mkremoteflash: Remote FlashCopy volume pair 2000:2200 successfully created. Use the lsremoteflash command to
determine copy completion.
dscli>
dscli> mkremoteflash -tgtinhibit -nocp -record -conduit IBM.2107-7520781/11 -dev IBM.2107-75ABTV1 2100-2101:2300-2301
dscli>
==========================================================================================================================================
2000:2200 20 0 Disabled Enabled Enabled Disabled Enabled Disabled Disabled 61036
dscli>
==========================================================================================================================================
dscli>
24.1.6 Start Global Mirror

To start Global Mirror operation, we must perform the following steps:
1. Define the Global Mirror session on the involved LSSs (master and subordinates).
2. Add the A volumes to the session.
3. Start the Global Mirror session.
Define the Global Mirror session on the involved LSSs

The mksession command defines the Global Mirror session to the specified LSSs. You do
this to all the LSSs that are going to be involved in the Global Mirror session, master and
subordinates. You can verify the results with the lssession command.
In our example, we have two LSSs in the local DS8000 that are going to participate in the
Global Mirror environment, LSS10 and LSS11. Therefore, we give the mksession command
twice and in each occasion we use the -lss parameter to specify the selected LSS. You also

specify the Global Mirror session ID with this command. This session ID is used when you
start Global Mirror in a later step. In our example we specify 02 for the Global Mirror session
ID. Example 24-3 shows the commands mksession and lssession we used in our example.
Example 24-3 Open the Global Mirror session on each LSS

dscli> mksession -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 02
CMUC00145I mksession: Session 02 opened successfully.
dscli>
dscli> mksession -dev IBM.2107-7520781 -lss IBM.2107-7520781/11 02
dscli>
dscli> lssession -l IBM.2107-7520781/10
LSS ID Session Status Volume VolumeStatus PrimaryStatus SecondaryStatus FirstPassComplete AllowCascading
========================================================================================================
10 02 - - - - - - -
dscli>
========================================================================================================
11 02 - - - - - - -
dscli>
Add the A volumes to the session

The next step is to add the A volumes to the session that was defined in the previous step. For
this we use the chsession -action add -volume command, and we verify the results with the
lssession command. Example 24-4 shows the commands we used in our example.
Example 24-4 Add the A volumes to the session on each LSS

dscli> chsession -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -action add -volume 1000-1001 02
CMUC00147I chsession: Session 02 successfully modified.
dscli>
dscli> chsession -dev IBM.2107-7520781 -lss IBM.2107-7520781/11 -action add -volume 1100-1101 02
dscli>
=================================================================================================================
10 02 Normal 1000 Join Pending Primary Copy Pending Secondary Simplex True Disable
dscli>
=================================================================================================================
With the chsession command you specify the Global Mirror session ID (02 in our example)
and the volumes that are going to be part of the Global Mirror environment. After adding the A
volumes to the session, the lssession command shows the volumes ID with its state in the
LSS. At this time of the procedure, the volumes state is join pending. This means that the
volumes are not active for the current session. We have not yet started the Global Mirror
session.
Note: At this step we do not have to do anything to add the B and C volumes to the Global
Mirror session. They are automatically recognized by the Global Mirror mechanism through
the Global Copy relationships and the FlashCopy relationships.
In addition to the chsession command, you can also add the A volumes with the mksession
command when you define the Global Mirror session on a LSS; see Example 24-5.
Example 24-5 Add the A volumes when you create a Global Mirror session
dscli> mksession -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -volume 1000-1001 02
dscli>
dscli> mksession -dev IBM.2107-7520781 -lss IBM.2107-7520781/11 -volume 1100-1101 02
dscli>
=================================================================================================================
dscli>
=================================================================================================================
Start the Global Mirror session - when there are no subordinates

Now we can start the Global Mirror session, so the process of Consistency Group formation
begins. For this we use the mkgmir command. The results can be verified with the showgmir
command; see Example 24-6.
Example 24-6 Start Global Mirror session 02

dscli> mkgmir -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -session 02
CMUC00162I mkgmir: Global Mirror for session 02 successfully started.
dscli>
dscli> showgmir -dev IBM.2107-7520781 IBM.2107-7520781/10
ID IBM.2107-7520781/10
Master Count 1
Master Session ID 0x02
Copy State Running
Fatal Reason Not Fatal
CG Interval (seconds) 0
XDC Interval(milliseconds) 50
CG Drain Time (seconds) 30
Current Time 11/10/2005 02:27:51 JST
CG Time 11/10/2005 02:27:50 JST
Successful CG Percentage 100
FlashCopy Sequence Number 0x43723196

Master ID IBM.2107-7520781
Subordinate Count 0
Master/Subordinate Assoc -
In the mkgmir command, the LSS we specified with the -lss parameter becomes the master. In
our example, it is LSS10. With this command we also specify the Global Mirror session ID of
the session we are starting.
At the time when you start the Global Mirror session, you can change the Global Mirror tuning
parameters of the session with the mkgmir command:
򐂰 -cginterval: Specifies how long to wait between the formation of Consistency Groups. If
this number is not specified or is set to zero, Consistency Groups are formed continuously.
򐂰 -coordinate: Indicates the maximum time that Global Mirror processing can hold host I/Os
in the source disk subsystem to start forming a Consistency Group.
򐂰 -drain: Specifies maximum amount of time in seconds allowed for the data to drain to the
remote site before failing the current Consistency Group.
For additional discussion about the tuning parameters refer to 20.4.2, “Consistency Group
parameters” on page 261 and Chapter 23, “Global Mirror performance and scalability” on
page 295.
This showgmir command shows the Global Mirror current status. The Copy State field
indicates Running, which means that Global Mirror is satisfactorily operating. A Fatal state
would have indicated that Global Mirror failed, and the Fatal Reason field would have shown
the reason for the failure.
The showgmir command also shows the current time in the Current Time field, which is the
time when the DS8000 received this command. The time when the last successful
Consistency Group was formed is shown in the CG Time field. You can obtain the current
Recovery Time Objective (RPO) for this Global Mirror session from the difference between
the Current Time and the CG Time.
From the lssession command in Example 24-7 you can see that after starting the Global
Mirror session, the VolumeStatus of the A volumes changes from Join Pending to Active.
Example 24-7 The A volumes status after starting the Global Mirror session
dscli> lssession 10-11
LSS ID Session Status Volume VolumeStatus PrimaryStatus SecondaryStatus FirstPassComplete AllowCascadin
========================================================================================================================
10 02 CG In Progress 1000 Active Primary Copy Pending Secondary Simplex True Disable
If we use the -metrics parameter with the showgmir command, we can obtain metrics for
Global Mirror after we have started the session; see Example 24-8.
Example 24-8 The showgmir command with -metrics parameter

dscli> showgmir -metrics -dev IBM.2107-7520781 IBM.2107-7520781/10
ID IBM.2107-7520781/10
Total Failed CG Count 0
Total Successful CG Count 317
Failed CG after Last Success 0
Last Successful CG Form Time 11/11/2005 17:28:49 JST
Coord. Time (seconds) 50
Interval Time (seconds) 0
Max Drain Time (seconds) 30
First Failure Control Unit -
First Failure LSS -
First Failure Status No Error
First Failure Reason -
First Failure Master State -
Last Failure Control Unit -
Last Failure LSS -
Last Failure Status No Error
Last Failure Reason -
Last Failure Master State -
Previous Failure Control Unit -
Previous Failure LSS -
Previous Failure Status No Error
Previous Failure Reason -
Previous Failure Master State -
In Example 24-8 on page 312, the Total Failed CG Count field indicates the total number of
Consistency Groups that did not complete successfully after we have started the Global
Mirror. The Total Successful CG Count indicates the total number of Consistency Groups that
completed successfully. First Failure indicates the first failure after we have started this
session. Last Failure indicates the latest failure, and Previous Failure indicates the failure
before the latest one. All this failure information will be cleared after we stop the Global Mirror
and start it again. Pausing and resuming the Global Mirror operation does not reset this
information.
Depending on the Global Mirror parameters you set and your system environment, the
Consistency Group formation can occasionally fail and the showgmir -metrics may show the
error messages. A typical case is that you see Max Drain Time Exceeded with the showgmir
command when data of the out-of-sync bitmap cannot be drained within the specified time.
However, this failure does not mean that you lose consistent data at the remote site, because
Global Mirror does not take FlashCopy (B to C) for the failed Consistency Group data. Global
Mirror will continue to attempt to form additional Consistency Groups without external
intervention. If failures repeatedly continue (no more Consistency Groups are formed), the
percentage of successful Consistency Groups is unacceptable (many failures occur), or the
frequency of Consistency Groups is not meeting your requirements (Recovery Point
Objective-RPO), then the failures are a problem and need to be addressed.
There is another command related to Global Mirror, which is the showgmiroos command. This
command reports the number of Out Of Sync Tracks that at a given moment Global Mirror has
to transmit to the remote site —the size of the logical track on the DS8000 FB volume is 64
KB. With the -scope parameter you select either the storage image scope or the LSS scope
for the information to be reported. See Example 24-9.
Example 24-9 The showgmiroos command

dscli> showgmiroos -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -scope si 02
Scope IBM.2107-7520781
Session 02
dscli>
dscli> showgmiroos -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -scope lss 02
Scope IBM.2107-7520781/10

Session 02
OutOfSyncTracks 303
dscli> showgmiroos -dev IBM.2107-7520781 -lss IBM.2107-7520781/11 -scope lss 02
Scope IBM.2107-7520781/11
Session 02
OutOfSyncTracks 0
Start the Global Mirror session - when there is a subordinate

If we are going to start a Global Mirror configuration where there is more than one storage
disk subsystem at the local site, which means more than one DS8000 (or DS8000 storage
image) at the local site, then the Global Mirror control path information between the master
LSS and the LSSs in the subordinate storage images must be indicated when we start the
Global Mirror session. Prior to this operation, we must have created the Global Mirror control
paths between the master LSS and the LSSs in the subordinates.
Figure 24-2 shows the example DS8000 configuration where we have a total of eight A
volumes (four on DS8000#1 and four DS8000#3) at the local site that participate in the Global
Mirror session. We have one DS8000 (DS8000#2) at the remote site with the corresponding
B and the C volumes.
Global Copy FlashCopy

A B C
Global Copy paths FlashCopy LSS22
LSS10 LSS20
GM Master
1000 2000 2200
Global Copy Pairs
1001 2001 2201
Global Mirror Control Path
LSS11 FCP links

LSS21 FlashCopy LSS23
1100 2100 2300

Global Copy Pairs
1101 2101 2301
DS8000#1
-dev IBM.2107-7520781
9000 2400 2600

Global Copy Pairs
9001 2401 2601
9100 2500 2700

Global Copy Pairs
9101 2501 2701
DS8000#3 DS8000#2
Figure 24-2 Start Global Mirror session with a subordinate DS8000#3
Example 24-10 shows how to start a Global Mirror configuration when there is a subordinate.
The example does not show how to set up the Global Copy and FlashCopy relationships
because these steps are exactly the same as in a no-subordinate situation.
Example 24-10 Start Global Mirror session when there is a subordinate

<< Create Global Mirror control path between DS8000#1 and DS8000#3 >>
dscli> lsavailpprcport -l -remotedev IBM.2107-7503461 -remotewwnn 5005076303FFC08F 10:90
===================================================
dscli>
dscli> mkpprcpath -remotedev IBM.2107-7503461 -remotewwnn 5005076303FFC08F -srclss 10 -tgtlss 90
i0001:i0031 i0101:i0101
dscli>
dscli> lspprcpath -fullid 10
===================================================================================================================
IBM.2107-7520781/10 IBM.2107-7503461/90 Success FF90 IBM.2107-7520781/I0001 IBM.2107-7503461/I0031 5005076303FFC08F
IBM.2107-7520781/10 IBM.2107-7503461/90 Success FF90 IBM.2107-7520781/I0101 IBM.2107-7503461/I0101 5005076303FFC08F
<< Setup Global Mirror environment bewteen DS8000#3 and DS8000#2 (These steps are NOT shown here) >>
<< Start Global Mirror with a Subordinate (DS8000#3) >>

IBM.2107-7520781/10:IBM.2107-7503461/90
dscli>
ID IBM.2107-7520781/10
Master Count 1
Copy State Running
Current Time 11/10/2005 21:08:42 JST
CG Time 11/10/2005 21:08:42 JST
FlashCopy Sequence Number 0x4373384A
Subordinate Count 1
Master/Subordinate Assoc IBM.2107-7520781/10:IBM.2107-7503461/90
24.2 Remove a Global Mirror environment with the DS CLI

In this section we provide an example of how to remove a Global Mirror environment using the
DS Command-Line Interface.

The Global Mirror environment delete process can be structured in five consecutive steps:
1. End Global Mirror processing.
2. Remove volumes from the session.
3. Remove the Global Mirror session.
4. Terminate the FlashCopy pairs.
5. Terminate the Global Copy pairs.
6. Remove paths.
– Between the local site to the remote site
– Between the master LSS and the subordinate LSSs
24.2.1 End Global Mirror processing

We terminate Global Mirror processing with the rmgmir command. With this command, we
have to specify the master LSS and the Global Mirror session ID of the session we are going
to close; see Example 24-11. Before you end Global Mirror processing, first display the
session information using the showgmir command.
You can use the -quiet parameter to turn off the confirmation prompt for this command.
Example 24-11 Terminate Global Mirror

ID IBM.2107-7520781/10
Master Count 1
Copy State Running
Current Time 11/10/2005 22:16:11 JST
CG Time 11/10/2005 22:16:10 JST
FlashCopy Sequence Number 0x4373481A
Subordinate Count 0
dscli>
dscli> rmgmir -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -session 02
CMUC00166W rmgmir: Are you sure you want to stop the Global Mirror session 02:? [y/n]:y
CMUC00165I rmgmir: Global Mirror for session 02 successfully stopped.
dscli>
ID IBM.2107-7520781/10
Master Count -
Master Session ID -
Copy State -
Fatal Reason -
CG Interval (seconds) -
XDC Interval(milliseconds) -
CG Drain Time (seconds) -
Current Time -
CG Time -
Successful CG Percentage -
FlashCopy Sequence Number -
Master ID -
Subordinate Count -
Example 24-11 on page 316 illustrates how to end Global Mirror session 02 processing.
Although this command might interrupt the formation of a consistency group, every attempt is
made to preserve the previous consistent copy of the data on the FlashCopy target volumes.
If, due to failures, this command cannot complete without compromising the consistent copy,
the command stops processing and an error code is issued. If this occurs, reissue the rmgmir
command with the -force parameter to force the command to stop the Global Mirror process.
If the Global Mirror configuration that you started also had subordinates, then to end Global
Mirror you also have to specify the Global Mirror control path information with the rmgmir
command. Otherwise, the command fails; see Example 24-12.
Example 24-12 Terminate Global Mirror when there is a subordinate

CMUN03067E rmgmir: Copy Services operation failure: configuration does not exist
dscli>
IBM.2107-7520781/10:IBM.2107-7503461/90
24.2.2 Remove the A volumes from the Global Mirror session

With the chsession command with the -action remove -volume parameters you remove the A
volumes from the Global Mirror session for a given LSS; see Example 24-13. First, you give
an lssession command to get Global Mirror session volumes information for the required
LSSs.
Example 24-13 Remove the A volumes from the Global Mirror session
LSS ID Session Status Volume VolumeStatus PrimaryStatus SecondaryStatus FirstPassComplete AllowCas
===========================================================================================================
dscli>
dscli> chsession -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -action remove -volume 1000-1001 02
dscli>
dscli> chsession -dev IBM.2107-7520781 -lss IBM.2107-7520781/11 -action remove -volume 1100-1101 02
dscli>

========================================================================================================
10 02 - - - - - - -
11 02 - - - - - - -
24.2.3 Remove the Global Mirror session

With the rmsession command you un-define the Global Mirror session from an LSS; see
Example 24-14. You do this to all the source LSSs where the Global Mirror session was
defined. At the end, you do an lssession command to verify that the Global Mirror session
has been removed.
Example 24-14 Remove the Global Mirror session from the LSSs
========================================================================================================
10 02 - - - - - - -
11 02 - - - - - - -
dscli>
dscli> rmsession -quiet -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 02
CMUC00146I rmsession: Session 02 closed successfully.
dscli>
dscli> rmsession -quiet -dev IBM.2107-7520781 -lss IBM.2107-7520781/11 02
dscli>
CMUC00234I lssession: No Session found.
Before closing the Global Mirror session on the LSS, we must remove all the A volumes from
the Global Mirror session on that LSS, otherwise, the rmsession command fails; see
Example 24-15.
Example 24-15 rmsession command fails when Global Mirror volumes were not previously removed
===========================================================================================================
dscli>
dscli> rmsession -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 02
CMUC00148W rmsession: Are you sure you want to close session 02? [y/n]:y
CMUN03107E rmsession: Copy Services operation failure: volumes in session
24.2.4 Terminate FlashCopy pairs
Depending on your network environment, you can give the FlashCopy commands to the local
disk subsystem for its inband transmission to the remote disk subsystem. In this case you use
the rmremoteflash command. Alternatively, if you have connectivity to the remote disk
subsystem, then you can give the rmflash command directly to the remote disk subsystem.
In our example we use the inband functionality of FlashCopy, in which case we have to
specify the LSS having the A volume for the -conduit parameter and the storage image ID at
the remote site for the -dev parameter; see Example 24-16. You have to give this command to
the DS HMC connected to the local DS8000#1.
Before terminating the pairs, we gather information with a lsremoteflash command.
Example 24-16 Remove all FlashCopy relationships between the B and C volumes
========================================================================================================================
2000:2200 20 43734829 Disabled Enabled Enabled Disabled Enabled Disabled Disabled
dscli>
========================================================================================================================
dscli>
dscli> rmremoteflash -quiet -conduit IBM.2107-7520781/10 -dev IBM.2107-75ABTV1 2000-2001:2200-2201
CMUC00180I rmremoteflash: Removal of the remote FlashCopy volume pair 2000:2200 has been initiated successfully. Use the
lsremoteflash command to determine when the relationship is deleted.
dscli>
dscli> rmremoteflash -quiet -conduit IBM.2107-7520781/11 -dev IBM.2107-75ABTV1 2100-2101:2300-2301
dscli>
CMUC00234I lsremoteflash: No Remote Flash Copy found.
dscli>
24.2.5 Terminate Global Copy pairs and remove the paths

For the termination of the Global Copy pairs you use the rmpprc command, and for the
deletion of the paths you use the rmpprcpath command; see Example 24-17 on page 320.
Before terminating the pairs and deleting the paths you request information by means of the
lspprc and the lspprcpath commands, respectively.
Note that the tasks you must perform to terminate the Global Copy relationships in a Global
Mirror environment are similar to what has been presented in Part 5, “Global Copy” on

page 185. You can refer to 19.2, “Remove Global Copy environment using DS CLI” on
page 217.
Example 24-17 Remove all Global Copy pairs and remove the paths
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
dscli> rmpprc -remotedev IBM.2107-75ABTV1 -quiet 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
=========================================================
10 20 Success FF20 I0143 I0010 5005076303FFC663
10 20 Success FF20 I0213 I0140 5005076303FFC663
11 21 Success FF21 I0143 I0010 5005076303FFC663
11 21 Success FF21 I0213 I0140 5005076303FFC663
dscli>
24.3 Manage the Global Mirror environment with the DS CLI

In this section we discuss and give examples of how to perform common Global Mirror control
tasks using the DS CLI. The following management activities are presented:
򐂰 Pause and resume the Global Mirror Consistency Group formation.
򐂰 Change the Global Mirror tuning parameters.
򐂰 Stop and start Global Mirror.
򐂰 Add and remove A volumes to the Global Mirror environment.
򐂰 Add and remove an LSS to an existing Global Mirror environment.
򐂰 Add and remove a subordinate disk subsystem to an existing Global Mirror environment.
24.3.1 Pause and resume Global Mirror Consistency Group formation

The pausegmir command pauses Global Mirror Consistency Group formation. You have to
specify the Global Mirror master LSS ID and session ID. You can verify the result with the
showgmir command, which shows Paused state in the Copy State field; see Example 24-18 on
page 321.
Note: The pausegmir command does not influence the Global Copy data transfer.
When you pause a Global Mirror session with the pausegmir command it will complete the
Consistency Group formation in progress before it pauses. This is slightly different from the
usage of the rmgmir command; refer to 24.3.3, “Stop and start Global Mirror” on page 324.
Example 24-18 Pause Global Mirror CG formation

ID IBM.2107-7520781/10
Master Count 1
Copy State Running
Current Time 11/11/2005 00:18:37 JST
CG Time 11/11/2005 00:18:37 JST
FlashCopy Sequence Number 0x437364CD
Subordinate Count 0
dscli>
LSS ID Session Status Volume VolumeStatus PrimaryStatus SecondaryStatus FirstPassComplete AllowCasca
=====================================================================================================================
dscli>
dscli> pausegmir -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -session 02
CMUC00163I pausegmir: Global Mirror for session 02 successfully paused.
dscli>
ID IBM.2107-7520781/10
Master Count 1
Copy State Paused
Current Time 11/11/2005 00:25:19 JST
CG Time 11/11/2005 00:25:10 JST
Subordinate Count 0
dscli>
=================================================================================================================
10 02 Normal 1000 Active Primary Copy Pending Secondary Simplex True Disable

dscli>
========================================================================================================================
dscli>
========================================================================================================================
dscli>
<< SequnceNum field does not change >>

========================================================================================================================
dscli>
========================================================================================================================
dscli>
The Status shown by the lssession command changes from CG In Progress, which means
that the Consistency Group of the session is in progress, to Normal, which means that the
session is in a normal Global Copy state. In fact, you see this state (Normal) between the
time when a FlashCopy has been taken and the next Global Copy Consistency Group
formation time.
Note: The FlashCopy sequence number has not changed after pausing Global Mirror
because the FlashCopy at the remote site has never been done; see Example 24-18.
The resumegmir command resumes Global Mirror processing for a specified session; see
Example 24-19. Consistency Group formation is resumed.
Example 24-19 Resume Global Mirror processing

dscli> resumegmir -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -session 02
CMUC00164I resumegmir: Global Mirror for session 02 successfully resumed.
dscli>
ID IBM.2107-7520781/10
Master Count 1
Copy State Running
Current Time 11/11/2005 01:20:41 JST
CG Time 11/11/2005 01:20:41 JST
Subordinate Count 0
24.3.2 Change the Global Mirror tuning parameters

You can change the three Global Mirror tuning parameters by pausing and resuming Global
Mirror. In Example 24-20 we changed the Consistency Group interval time parameter from
zero to 60 seconds.
Example 24-20 Change the Consistent Group interval (CG Interval) time parameter
ID IBM.2107-7520781/10
Master Count 1
Copy State Running
Current Time 11/11/2005 01:20:41 JST
CG Time 11/11/2005 01:20:41 JST
Subordinate Count 0
dscli>
dscli>
dscli> resumegmir -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -session 02 -cginterval 60
dscli>
ID IBM.2107-7520781/10
Master Count 1
Copy State Running
Current Time 11/11/2005 01:28:29 JST
CG Time 11/11/2005 01:28:25 JST
Subordinate Count 0

24.3.3 Stop and start Global Mirror
To stop Global Mirror means to stop the Global Mirror master on a LSS, using the rmgmir
command; see Example 24-21. You will stop Global Mirror when, for example, you want to
start using a different topology; see 21.3.4, “Global Mirror environment topology changes” on
page 269.
You do not need to remove the Global Copy and FlashCopy relationships to stop the Global
Mirror master. After stopping the Global Mirror master, the Consistency Group formation does
not continue, which means the FlashCopy sequence number at the remote site does not
increment.
Although the operation to stop Global Mirror with the rmgmir command might interrupt the
formation of a Consistency Group, every attempt is made to preserve the previous consistent
copy of the data on the FlashCopy target volumes. If due to failures the rmgmir command
cannot complete without compromising the consistent copy, the command stops processing
and an error code is issued. If this occurs, reissue the rmgmir command with the -force
parameter to force the command to stop the Global Mirror process.
The mkgmir command restarts the Global Mirror master. You can specify another LSS on
which you will start the Global Mirror master. In Example 24-21, we stop the Global Mirror
master running on LSS10 and then start again on LSS11.
Example 24-21 Stop and start Global Mirror

========================================================================================================================
dscli>
dscli> rmgmir -quiet -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -session 02
dscli>
ID IBM.2107-7520781/10
Master Count -
Master Session ID -
Copy State -
Fatal Reason -
CG Interval (seconds) -
XDC Interval(milliseconds) -
CG Drain Time (seconds) -
Current Time -
CG Time -
Successful CG Percentage -
Master ID -
Subordinate Count -
dscli>
=================================================================================================================
dscli>
dscli>
========================================================================================================================
dscli>
ID IBM.2107-7520781/11
Master Count 1
Copy State Running
Current Time 11/11/2005 21:49:09 JST
CG Time 11/11/2005 21:49:08 JST
Subordinate Count 0
24.3.4 Add and remove A volumes to the Global Mirror environment

First you create the Global Copy (A to B) and FlashCopy (B to C) relationships for the A
volume that you want to add to the Global Mirror environment. After this, you add the A
volume to the Global Mirror session, using the chsession -action add -volume command. In
Example 24-22 we added A volume 1002, which is associated to B volume 2002 and C
volume 2202.
Example 24-22 Add an A volume to the Global Mirror environment

<< Preparing an A volume >>
dscli> mkpprc -remotedev IBM.2107-75ABTV1 -type gcp 1002:2002
dscli>
dscli> lspprc -l 1002
========================================================================================================================
dscli>
========================================================================================================================
dscli>
dscli> mkremoteflash -tgtinhibit -nocp -record -conduit IBM.2107-7520781/10 -dev IBM.2107-75ABTV1 2002:2202

lsremoteflash command to determine copy completion
dscli>
dscli> lsremoteflash -conduit IBM.2107-7520781/10 -dev IBM.2107-75ABTV1 2002:2202
ID SrcLSS SequenceNum ActiveCopy Recording Persistent Revertible SourceWriteEnabled TargetWriteEnabl
===========================================================================================================
2002:2202 20 0 Disabled Enabled Enabled Disabled Enabled Disabled
dscli>
<< Add the A volume to Global Mirror >>

dscli> lssession 10
===========================================================================================================
dscli>
dscli> chsession -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -action add -volume 1002 02
dscli>
dscli> lssession 10
===========================================================================================================
dscli>
dscli> lssession 10
===========================================================================================================
To be added to a Global Mirror session the A volumes may be in any state, such as simplex
(no relationship), copy pending, or suspended. Volumes that have not completed their initial
copy phase (also called first pass) stay in a Join Pending state until the first pass is complete.
You can check the first pass status with the lspprc -l command; see Example 24-23. The
First Pass Status field reports this information, where True means that the Global Copy first
pass has been completed.
Example 24-23 Check the first pass completion for the Global Copy initial copy
dscli> lspprc -l -fmt stanza 1002
ID 1002:2002
State Copy Pending
Reason -
Type Global Copy
Out Of Sync Tracks 0
Tgt Read Disabled
Src Cascade Disabled
Tgt Cascade invalid
Date Suspended -
SourceLSS 10
Timeout (secs) unknown
Critical Mode Disabled
First Pass Status True
When you want to remove an A volume from the Global Mirror environment, you can use the
chsession -action remove -volume command. You first remove the A volume from the
Global Mirror session and then remove its Global Copy and FlashCopy relationships. See
Example 24-24.
Example 24-24 Remove an A volume from the Global Mirror environment

========================================================================================================================
dscli>
dscli> chsession -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -action remove -volume 1002 02
dscli>
========================================================================================================================
Attention: Suspending or removing even one Global Copy pair that belongs to an active
Global Mirror session will impact the formation of Consistency Groups. If you suspend or
remove the Global Copy relationship from the A volume without removing the volume from
the Global Mirror session, it will cause Consistency Group formation to fail, and periodical
SNMP alerts will be issued.
24.3.5 Add and remove an LSS to an existing Global Mirror environment

First you create the Global Copy (A to B) and FlashCopy (B to C) relationships for the LSS,
and for the A volumes in the LSS that you want to add to the Global Mirror environment.
After this, you add the LSS with the mksession command, and then you add the A volume.
You can also use the mksession command to add the LSS along with the A volume; see
Example 24-25.
Example 24-25 Add an LSS to the Global Mirror session

<< Prepare the A volume >>
dscli> mkpprcpath -remotedev IBM.2107-75ABTV1 -remotewwnn 5005076303FFC663 -srclss 12 -tgtlss 24 i0143:i0010 i0213:i0140
dscli>
dscli> lspprcpath -fullid 12
===================================================================================================================

dscli>
dscli> mkpprc -remotedev IBM.2107-75ABTV1 -type gcp 1200:2400
dscli>
========================================================================================================================
dscli>
========================================================================================================================
dscli>
CMUN03044E mkremoteflash: 2400:2600: Copy Services operation failure: path not available
dscli>
dscli>
dscli> lsremoteflash -l -conduit IBM.2107-7520781/10 -dev IBM.2107-75ABTV1 2400
dscli>
dscli> lsremoteflash -l -conduit IBM.2107-7520781/12 -dev IBM.2107-75ABTV1 2400
==========================================================================================================================================
dscli>
<< Add a LSS and an A volume to the Global Mirror >>

dscli> lssession 12
CMUC00234I lssession: No Session found.
dscli>
dscli> mksession -dev IBM.2107-7520781 -lss IBM.2107-7520781/12 -volume 1200 02
dscli>
dscli> lssession 12
=================================================================================================================
dscli>
=================================================================================================================
When you remove an LSS from a Global Mirror environment, you have to first remove all the
A volumes on the LSS with the chsession command and then remove the LSS with the
rmsession command; see Example 24-26.
Example 24-26 Remove an LSS from a Global Mirror session

========================================================================================================================
dscli>
dscli> chsession -dev IBM.2107-7520781 -lss IBM.2107-7520781/12 -action remove -volume 1200 02
dscli>
========================================================================================================================
12 02 - - - - - - -
dscli>
dscli> rmsession -dev IBM.2107-7520781 -lss IBM.2107-7520781/12 02
CMUC00148W rmsession: Are you sure you want to close session 02? [y/n]:y
dscli>
========================================================================================================================
24.3.6 Add and remove a subordinate disk subsystem

In order to add or remove a subordinate storage disk subsystem from an existing Global
Mirror environment, you have to stop the Global Mirror session and start it again with or
without the subordinate specification.
This task is in fact a topology change of the Global Mirror configuration, which requires that
you stop Global Mirror first in order to re-start it again with the new configuration; see 21.3.4,
“Global Mirror environment topology changes” on page 269.
For examples of Global Mirror stop and start tasks see 24.3.3, “Stop and start Global Mirror”
on page 324.

24.4 Recovery scenario after local site failure with the DS CLI
The example presented in this section discusses how to perform the required steps to recover
from a production site failure using DS CLI commands. For a detailed discussion of the
internal Global Mirror considerations, refer to 21.5, “Recovery scenario after production site
failure” on page 274.
For this example we use the configuration that we set up in 24.1, “Set up a Global Mirror
environment using the DS CLI” on page 306.
Figure 24-3 shows the configuration during normal operations.
Application server and Application Backup server and

DS CLI client DS CLI client

GM Master
1000 2000 2200
Global Copy Pair
1001 2001 2201
1100 2100 2300

Global Copy Pair
1101 2101 2301
A B C
GC: Source GC: Target FC: Target
Copy Pending Copy Pending
FC: Source
DS8000#1 DS8000#2
Figure 24-3 Global Mirror example before unplanned production site failure
Summary of the recovery scenario

The typical recovery scenario after the production site failure is:
1. Stop Global Mirror processing.
2. Perform Global Copy Failover from B to A.
3. Verify for valid Consistency Group state.
4. Create consistent data on B volumes (Reverse FlashCopy from B to C).
5. Re-establish the FlashCopy relationship from B to C.
6. Restart application at remote site.
The following sections discuss each of the listed tasks of the recovery scenario.
24.4.1 Stop Global Mirror processing

Depending on the state of the Global Mirror local disk subsystem where the master was
running, you may be able to stop the Global Mirror session. The rmgmir command stops
Global Mirror processing. You give this command to the DS HMC connected to the local
DS8000 (DS8000#1); see Example 24-27 on page 331.
Example 24-27 Terminate Global Mirror
24.4.2 Perform Global Copy Failover from B to A

A Failover (Copy Services Failover function) on the Global Copy target B volumes turns these
volumes into source volumes and also suspends them immediately. You can use the
failoverpprc command to do this.
This Failover operation sets the stage for change recording when application updates start
changing the B volumes. Change recording in turn allows you to re-synchronize just the
changes from the B to the A volumes, later before returning to the local site. But at this stage,
the B volumes do not contain consistent data and are still useless. We just changed their
Global Copy state from target to source and suspended. Figure 24-4 shows the DS8000
environment after the Failover operation.

failoverpprc B to A
1000 2000 2200

Global Copy Pair
1001 2001 2201
1100 2100 2300

Global Copy Pair
1101 2101 2301
A B C
GC: Source GC: Source FC: Target
Copy Pending Suspended
FC: Source
DS8000#1 DS8000#2
Figure 24-4 Site swap scenario after failoverpprc
Example 24-28 shows the command for this operation. You can check the result with the
lspprc command.
Example 24-28 failoverpprc command example

dscli> lspprc 2000-2001 2100-2101
=========================================================================================================

dscli>
dscli> failoverpprc -remotedev IBM.2107-7520781 -type gcp 2000-2001:1000-1001 2100-2101:1100-1101
dscli>
dscli> lspprc 2000-2001 2100-2101
====================================================================================================
24.4.3 Verify for valid Consistency Group state

Now you have to investigate whether all FlashCopy relationships are in a consistent state.
This means that you have to query all FlashCopy relationships between B and C, which are
part of the Consistency Group, to determine the state of the FlashCopy relationship. Global
Mirror might have been in the middle of forming a Consistency Group and FlashCopy may
have not completed the creation of a complete set of consistent C volumes.
Each FlashCopy pair needs a FlashCopy query to identify its state. You use the lsflash
command to check the SequenceNum and the Revertible field status. Example 24-29 shows
the Revertible status of all of the FlashCopy pairs is Disabled (that is, non-revertible) and the
SequenceNums of all relationships are equal. Therefore, you do not need to take any action in
this case.
You can find a detailed discussion of this verification process in 21.5.4, “Verify for valid
Consistency Group state” on page 277.
Example 24-29 Verify the FlashCopy state

dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
2000:2200 20 4374ABB7 300 Disabled Enabled Enabled Disabled Enabled Disabled Disabled
Example 24-30 shows a hypothetical situation where all FlashCopy relationships are in the
revertible state and have the same sequence number. In this case, you should execute a
revertflash to all the FlashCopy relationships.
Example 24-30 All revertible and SeqNum is equal, then revertflash

dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
2000:2200 20 437895B2 300 Disabled Enabled Enabled Enabled Disabled Disabled Disabled
dscli>
dscli> revertflash 2000-2001 2100-2101
dscli>
dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
2000:2200 20 437895B2 300 Disabled Enabled Enabled Disabled Enabled Disabled Disabled
dscli>
If some FlashCopy pairs are revertible and others are not revertible while their sequence
numbers are equal, you should execute a commitflash command to the FlashCopy
relationships that have the revertible status; see Example 24-31.
When the FlashCopy relationship is not in a revertible state, the commit operation is not
possible. When you give this command to FlashCopy pairs that are non-revertible, you are
going to see only an error message, but no action is performed. To make the task easier, you
may run a commitflash command to all FlashCopy pairs. In Example 24-31, 2000 and 2001
are not in the revertible state; therefore, we see error messages.
Example 24-31 Some are revertible and SeqNum are equal, then commitflash
dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
dscli>
dscli> commitflash 2000-2001 2100-2101
CMUN03054E commitflash: 2000:2000: Copy Services operation failure: invalid revertible specification
CMUN03054E commitflash: 2001:2001: Copy Services operation failure: invalid revertible specification
dscli>
dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
Example 24-32 on page 334 shows a hypothetical situation where some FlashCopy pairs are
revertible and others are non-revertible. The revertible FlashCopy pairs' sequence numbers
are equal. The non-revertible FlashCopy pairs’ sequence numbers are also equal, but do not
match the revertible FlashCopies sequence number. In this case, you should give a
commitflash command to the FlashCopy relationships that have the revertible status.
When the FlashCopy relationship is non-revertible, the revert operation is not possible. When
you execute this command against FlashCopy pairs that are non-revertible, you are going to
see only an error message, but no action is performed. To make the task easier, you may run

a revertflash command to all FlashCopy pairs. In Example 24-32, 2000 and 2001 are not in
the revertible state. Therefore we see error messages.
Example 24-32 Some are revertible and SeqNum are not equal, then revertflash
dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
dscli>
dscli> revertflash 2000-2001:2200-2201 2100-2101:2300-2301
CMUC00083E revertflash: Invalid volume 2000-2001:2200-2201.
dscli>
dscli> revertflash 2000-2001 2100-2101
CMUN03054E revertflash: 2000:2000: Copy Services operation failure: invalid revertible specification
CMUN03054E revertflash: 2001:2001: Copy Services operation failure: invalid revertible specification
dscli>
dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
After these actions, all FlashCopy pairs are non-revertible and all sequence numbers are
equal, so now you can proceed to the next step.
24.4.4 Reverse FlashCopy from B to C

At this point only the C volumes (logical data) comprise a set of consistent data volumes,
although the physical data of the C volumes may be spread over the physical B and C
volumes. The B volumes (logical data) do not provide consistent data volumes because
Global Copy does not provide data consistency.
We want to have to two good copies of the data at the recovery site. The aim is to have a
consistent set of volumes to work with, still keeping a good copy to which we can resort to if
needed. The next step is then to create the same consistency on the B volumes as we have
on the C volumes; see Figure 24-5 on page 335. This can be achieved with the reverseflash
-fast command. This operation is called Fast Reverse Restore (FRR). You have to use the
-tgtpprc parameter with the reverseflash -fast command because the B volume is also a
Global Copy source at this step.
Note: Though the Fast Reverse Restore operation starts a background copy from the C to
the B volumes, in the reverseflash command you have to specify the B volumes as the
FlashCopy sources and the C volumes as the FlashCopy targets.
Figure 24-5 shows the remote DS8000 environment after the reverseflash command was
issued. After executing this command and before C to B background copy is completed, the C
volumes become the FlashCopy source and the B volumes become the FlashCopy target.

reverseflash B to C
LSS10 LSS20 Background LSS22

Copy
1000 2000 2200
Global Copy Pair
1001 2001 2201
LSS11 LSS21 Background LSS23
Copy
1100 2100 2300
Global Copy Pair
1101 2101 2301
A B C
GC: Source GC: Source FC: Source
FC: Target
DS8000#1 DS8000#2
Figure 24-5 Site swap scenario after the reverseflash command was issued
Example 24-33 shows the results of the reverseflash command. The lsflash command
shows volume 2200 (C volume) as the FlashCopy source.
Example 24-33 The reverseflash B to C

dscli> reverseflash -fast -tgtpprc 2000-2001:2200-2201 2100-2101:2300-2301
dscli>
dscli> lsflash -l 2000-2001 2100-2101
ID SrcLSS SequenceNum Timeout ActiveCopy Recording Persistent Revertible SourceWriteEnabled TargetWriteEnabled BackgroundCopy OutOfSyncTracks DateCreated
================================================================================================================================================================
2200:2000 22 4374ABB7 300 Enabled Disabled Disabled Disabled Enabled Enabled Enabled 40102 Fri Nov 11
dscli>
dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
2200:2000 22 4374ABB7 300 Enabled Disabled Disabled Disabled Enabled Enabled Enabled
dscli>

The above Fast Reverse Restore (FRR) operation does a background copy of all tracks that
changed on the B volumes since the last CG formation. This results in the B volumes
becoming equal to the image that was present on the C volumes. This is the logical view.
From the physical data placement point of view, the C volumes do not have meaningful data
after the FlashCopy relationship ends.
Because you do not specify the -persist parameter, the FlashCopy relationship ends after the
background copy from C to B completes; see Figure 24-6.

LSS10 LSS20 LSS22
1000 2000 2200

Global Copy Pair
1001 2001 2201
LSS11 LSS21 LSS23
1100 2100 2300

Global Copy Pair
1101 2101 2301
A B C
GC: Source GC: Source FC: None
FC: None
DS8000#1 DS8000#2
Figure 24-6 After the completion of the background copy
You have to wait until all Fast Reverse Restore operations and their background copy
complete successfully before you proceed with the next step. Again, when the background
copy completes, the FlashCopy relation will end. Therefore, you should check if the
FlashCopy relationships remain to determine when all Fast Reverse Restore operations have
completed; see Example 24-34. This example shows the result of the lsflash command after
the reverseflash background copy completes.
Example 24-34 The lsflash command to confirm the completion of the background copy
dscli> lsflash 2000-2001 2100-2101
CMUC00234I lsflash: No Flash Copy found.
24.4.5 Re-establish the FlashCopy relationship from B to C

mkflash B to C
LSS10 LSS20
FlashCopy LSS22
1000 2000 2200

Global Copy Pair
1001 2001 2201
LSS11 FlashCopy
LSS21 LSS23
1100 2100 2300

Global Copy Pair
1101 2101 2301
A B C
FC: Source
DS8000#1 DS8000#2
Figure 24-7 After mkflash B to C
In this step you create the former FlashCopy relationship between the B and C volumes, as
they were at the beginning when you set up the Global Mirror environment; see Figure 24-7.
This step is in preparation for returning later to production at the local site. The mkflash
command used in this step is illustrated in Example 24-35, and is exactly the same
FlashCopy command you may have used when you initially created the Global Mirror
environment. See 20.3.4, “Introduce FlashCopy” on page 256.
In a disaster situation it may be that you do not want to use the -nocp option for the FlashCopy
from B to C. This will remove the FlashCopy I/O overhead when the application starts.
Example 24-35 Re-establish the FlashCopy relationships from B to C

dscli> mkflash -tgtinhibit -nocp -record 2000-2001:2200-2201 2100-2101:2300-2301
dscli>
dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
2000:2200 20 0 300 Disabled Enabled Enabled Disabled Enabled Disabled Disabled

24.4.6 Restart the application at the remote site
Depending on your operating system, it may be necessary to re-scan Fibre Channel devices
(to remove device objects for the recovery site volumes and recognize the new sources) and
mount the B volumes. And then start all applications at the recovery site.
Now that the application has started at the remote site, all the write I/Os to the new source
volumes (that is, the B volumes) are tracked in the bit-maps by the Failover function.
Figure 24-8 shows this environment.

Start Application
I/O
LSS10 LSS20
FlashCopy LSS22
1000 2000 2200

Global Copy Pair
1001 2001 2201
LSS11 FlashCopy
LSS21 LSS23
1100 2100 2300

Global Copy Pair
1101 2101 2301
A B C
FC: Source
DS8000#1 DS8000#2
Figure 24-8 After the application start
24.5 Return to the local site

The return to the normal production site typically follows this scenario:
1. Create paths from B to A.
2. Perform Global Copy Failback from B to A.
3. Query for the Global Copy first pass completion.
4. Quiesce the application at the remote site.
5. Query the Out Of Sync Tracks until it shows zero.
6. Create paths from A to B if they do not exist.
7. Perform Global Copy Failover from A to B.
8. Perform Global Copy Failback from A to B.
9. Start Global Mirror.
10.Start the application at the local site.
The following sections discuss each of the preceding steps.
24.5.1 Create paths from B to A
The local site is operational again. If the local site did not lose the data at the time when the
swap to the remote site occurred, then it is possible to re-synchronize the changed data from
B to A in preparation for returning to the local site.
Before doing this failback process, we need paths to be defined from B to A. For this task you
use the lsavailpprcport, mkpprcpath, and lspprcpath commands; see Example 24-36. You
give these commands to the DS HMC connected to the remote DS8000#2.
Example 24-36 Create paths from B to A

===================================================
dscli>
===================================================
dscli>
dscli> mkpprcpath -remotedev IBM.2107-7520781 -remotewwnn 5005076303FFC1A5 -srclss 20 -tgtlss 10
i0010:i0143 i0140:i0213
dscli>
dscli> mkpprcpath -remotedev IBM.2107-7520781 -remotewwnn 5005076303FFC1A5 -srclss 21 -tgtlss 11
i0010:i0143 i0140:i0213
dscli>
===================================================================================================================
24.5.2 Perform Global Copy Failback from B to A

After defining the paths from B to A, you use the failbackpprc command to re-synchronize
the changed data from B to A. The failbackpprc command is issued to the B volume as the
source and the A volume as the target; see Example 24-37 on page 340.
This process changes the A volume from its previous state (source) copy pending to target
copy pending; see Figure 24-9 on page 340. You have to add the -type gcp parameter with
the failbackpprc command to request Global Copy mode.

failbackpprc B to A Application
running
LSS10 LSS20
FlashCopy LSS22
1000 2000 2200

Global Copy Pair
1001 2001 2201
LSS11 FlashCopy
LSS21 LSS23
Changed data sent from B to A
1100 2100 2300
Global Copy Pair
1101 2101 2301
A B C
GC: Target GC: Source FC: Target
FC: Source
DS8000#1 DS8000#2
Figure 24-9 Site swap scenario after Global Copy Failback from B to A
The failbackpprc initialization mode re-synchronizes the volumes in this manner:

򐂰 If a volume at the production site is in simplex state (no relationship), all of the data for that
volume is sent from the recovery site to the production site.
򐂰 If a volume at the production site is in the copy pending or suspended state and without
changed tracks, only the modified data on the volume at the recovery site is sent to the
volume at the production site.
򐂰 If a volume at the production site is in a suspended state and has tracks on which data has
been written, the volume at the recovery site will discover which tracks were modified on
any site and send both the tracks changed on the production site and the tracks marked at
the recovery site.
The volume at the production site becomes a write-inhibited target volume. This action is
performed on an individual volume basis.
Example 24-37 shows the commands given in our example. First we listed the status of the B
volumes and then performed the Global Copy Failback operation. The command is given to
the DS HMC connected to the remote DS8000#2.
Example 24-37 Perform Global Copy Failback from B to A

<< Before the failbackpprc B to A >>
<< B volume status >>
dscli> lspprc 2000-2001 2100-2101
====================================================================================================
<< The failbackpprc B to A >>
dscli> failbackpprc -remotedev IBM.2107-7520781 -type gcp 2000-2001:1000-1001 2100-2101:1100-1101
<< B volume status >>

dscli> lspprc 2000-2001 2100-2101
==================================================================================================
<< A volume status >>

dscli> lspprc 1000-1001 1100-1101
=========================================================================================================
Notes on the failbackpprc command

If the server at the production site is still online and accessing the disk or a crash happens, so
that the SCSI persistent reserve is still set on the previous source disk, the failbackpprc
command fails. In this case, the server at the production site locks the target with a SCSI
persistent reserve. After this SCSI persistent reserve is reset with the varyoffvg command
(in this case on AIX), the failbackpprc command completes successfully.
There is a -resetreserve parameter for the failbackpprc command. This option resets the
reserved state so that the failback operation can complete. In the failback operation after a
real disaster, you may use this parameter because the server might go down while the SCSI
persistent reserve was set on the A volume. If the server at the production site is operational,
for example, when a Global Mirror failback operation test is performed, you must not use this
parameter because the server at the local site still owns the A volume and may be using it and
the failback operation suddenly changes the contents of the volume. This may cause the
server file system’s corruption.
24.5.3 Query for the Global Copy first pass completion

The first pass of Global Copy is the first phase of the re-synchronization process, when all the
data that has changed while the B volumes were suspended is copied to the A volumes. As
long as the first pass copy process continues, the Out Of Sync Tracks does not show zero.
Therefore, depending on your failback scenario, you can continue to run the application at the
remote site until the Global Copy first pass process completes.

You can query this status with the lspprc command; see Example 24-38. The First Pass
Status field indicates the status of the first pass, where True means that the first pass has
completed.
Example 24-38 Query for the Global Copy first pass completion
dscli> lspprc -l 2000-2001 2100-2101
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs) Critical Mode First Pass
Status
===============================================================================================================================================================
2000:1000 Copy Pending - Global Copy 0 Disabled Disabled invalid - 20 unknown Disabled True
dscli>
24.5.4 Quiesce the application at the remote site

Before returning to normal operation on the local site, the application (still updating B volumes
in the recovery site) must be quiesced to cease all write I/O from updating the B volumes.
Depending on the host operating system, it may be necessary to dismount the B volumes.
24.5.5 Query the Out Of Sync Tracks until it shows zero

After quiescing the application, to ensure that all the data has been written to the B volumes
you wait until the Out Of Sync Tracks for the Global Copy pairs shows zero. You can check
this status with the lspprc -l command; see Example 24-39. You have to issue this
command to the DS HMC connected to the remote DS8000 (DS8000#2).
Example 24-39 Query the Global Copy Out Of Sync Tracks until it shows zero
dscli> lspprc -l 2000-2001 2100-2101
ID State Reason Type OutOfSyncTracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs) Critical Mode First Pass
================================================================================================================================================================
dscli>
24.5.6 Create paths from A to B if they do not exist

Most likely there are no paths from A to B at this point. You can check the current paths status
with the lspprcpath command. We recommend that you run this command with the -fullid
command flag so that you get fully qualified IDs in the output report. The fully qualified ID
information will be of help when trying to identify whether you have paths between the correct
DS8000s (DS8000#1 to DS8000#2 in this example). You have to run this command on the
DS HMC connected to the local DS8000 (DS8000#1).
In our example, the paths were still available; see Example 24-40. In there were no available
paths, then you must define them now using the mkpprcpath command; Example 24-1 on
page 307 shows how to do this task.
Example 24-40 Check available paths from A to B

===================================================================================================================
24.5.7 Perform Global Copy Failover from A to B

In order to return to the original configuration we have to return the A volumes to their original
Global Copy (source) copy pending volume state. This is a two-step procedure. First, the
failoverpprc command converts the state of the A volumes from target copy pending to
(source) suspended. The state of the B volumes is preserved; see Figure 24-10.

failoverpprc A to B Application
running
LSS10 LSS20
FlashCopy LSS22
1000 2000 2200

Global Copy Pair
1001 2001 2201
LSS11 FlashCopy
LSS21 LSS23
1100 2100 2300

Global Copy Pair
1101 2101 2301
A B C
Suspended Copy Pending
FC: Source
DS8000#1 DS8000#2
Figure 24-10 Site swap scenario after Global Copy Failover from A to B
Example 24-41 shows the result of the failoverpprc command we used in our example, and
the volume state after this command is issued. You have to give this command to the DS
HMC connected to the local DS8000 (DS8000#1).
Example 24-41 Global Copy Failover from A to B

<< DS8000 #1 >>
dscli> failoverpprc -remotedev IBM.2107-75ABTV1 -type gcp 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
dscli> lspprc 1000-1001 1100-1101
====================================================================================================

<< DS8000 #2 >>

dscli> lspprc 2000-2001 2100-2101
==================================================================================================
24.5.8 Perform Global Copy Failback from A to B

Next we return the Global Copy pairs to the original configuration with the failbackpprc
command. Figure 24-11 shows the configuration after this command is executed.

failbackpprc A to B Application
running
LSS10 LSS20
FlashCopy LSS22
1000 2000 2200

Global Copy Pair
1001 2001 2201
LSS11 FlashCopy
LSS21 LSS23
1100 2100 2300

Global Copy Pair
1101 2101 2301
A B C
FC: Source
DS8000#1 DS8000#2
Figure 24-11 Site swap scenario after failbackpprc A to B
Example 24-42 shows the result of the failbackpprc command used in our example, and the
volume state after this command is issued. You have to give this command to the DS HMC
connected to the local DS8000 (DS8000#1).
Example 24-42 Global Copy Failback from A to B

<< DS8000 #1 >>
dscli> failbackpprc -remotedev IBM.2107-75ABTV1 -type gcp 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
<< DS8000 #2 >>

dscli> lspprc 2000-2001 2100-2101
=========================================================================================================
24.5.9 Start Global Mirror

mkgmir Application
running
LSS10 LSS20
FlashCopy LSS22
GM Master
Started 1000 2000 2200
Global Copy Pair
1001 2001 2201
LSS11 FlashCopy
LSS21 LSS23
1100 2100 2300

Global Copy Pair
1101 2101 2301
A B C
FC: Source
DS8000#1 DS8000#2
Figure 24-12 Start Global Mirror
The last step before starting the application at the production site is to start the Global Mirror
session again; see Figure 24-12, “Start Global Mirror” on page 345. If you did not already
create the FlashCopy relationships from B to C volumes, then you have to do it before starting
the Global Mirror.

To start the Global Mirror session use the mkgmir command. Before starting Global Mirror, you
can check the status for the Global Mirror session on each LSS with the lssession
command. After starting Global Mirror, you can use the showgmir command to check its
status.
Example 24-43 shows the commands used in our example and the corresponding results.
You run this command on the DS HMC connected to the local DS8000 (DS8000#1).
Example 24-43 Start Global Mirror

===========================================================================================================
dscli>
dscli>
ID IBM.2107-7520781/10
Master Count 1
Copy State Running
Current Time 11/12/2005 00:22:19 JST
CG Time 11/12/2005 00:22:19 JST
FlashCopy Sequence Number 0x4374B72B
Subordinate Count 0
24.5.10 Start the application at the local site
Now we have an environment on which to start the application at the original local site.
Depending on your operating system, it may be necessary to re-scan Fibre Channel devices
and mount the new source volumes (A volumes) at the local site. Start all applications and
check for consistency; see Figure 24-13. Depending on your path design, delete the paths
from the recovery to the production LSSs.

Start Application Application
running
I/O
LSS10 LSS20
FlashCopy LSS22
GM Master
1000 2000 2200
Global Copy Pair
1001 2001 2201
LSS11 FlashCopy
LSS21 LSS23
1100 2100 2300

Global Copy Pair
1101 2101 2301
A B C
FC: Source
DS8000#1 DS8000#2
Figure 24-13 Start the application at the local site
24.6 Practice disaster recovery readiness

In this section we discuss how to practice your disaster recovery readiness without stopping
the application at the production site. You can use the same procedure to make a test or
make a regular backup copy at the remote site. The typical scenario for this activity is the
following:
1. Query the Global Mirror environment to have a look at the situation.
2. Pause Global Mirror and check its completion.
3. Pause Global Copy pairs.
4. Perform Global Copy Failover from B to A.
5. Create consistent data on B volumes (reverse FlashCopy from B to C).
6. Wait for the FlashCopy background copy to complete.
7. Re-establish FlashCopy pairs — B to C with original Global Mirror options.
8. Take FlashCopy from B to D.
9. Perform the disaster recovery testing using the D volume.
10.Perform Global Copy Failback from A to B.
11.Resume Global Mirror.

Many steps in the above scenario are the same that were discussed in 24.4, “Recovery
scenario after local site failure with the DS CLI” on page 330, and 24.5, “Return to the local
site” on page 338. For those that are similar, we provide their pointers here.
24.6.1 Query the Global Mirror environment to look at the situation

There are several commands that you can use to look at the situation.
Query the Global Copy status

The lspprcpath and lspprc commands; see 24.1.4, “Create Global Copy relationships - A to
B volumes” on page 307.
Query the FlashCopy status

The lsremoteflash (or lsflash) command; see 24.1.5, “Create FlashCopy relationships - B
to C volumes” on page 308.
Query the Global Mirror status

The lssession, showgmir, showgmir -metrics, and showgmiroos commands; see 24.1.6,
“Start Global Mirror” on page 309.
24.6.2 Pause Global Mirror and check its completion

Example 24-44 shows how to perform this task. For detailed discussion and considerations
see 24.3.1, “Pause and resume Global Mirror Consistency Group formation” on page 320.
Give this command to the DS HMC connected to the local disk subsystem.
Example 24-44 Pause Global Mirror

dscli>
ID IBM.2107-7520781/10
Master Count 1
Copy State Paused
Current Time 11/14/2005 18:50:04 JST
CG Time 11/14/2005 18:49:59 JST
FlashCopy Sequence Number 0x43785DC7
Subordinate Count 0
dscli>
24.6.3 Pause Global Copy pairs
The pausepprc command suspends the Global Copy pairs; see Example 24-45. Give this
command to the DS HMC connected to the local disk subsystem.
Example 24-45 Pause Global Copy pairs

<< DS8000 #1 >>
dscli> pausepprc -remotedev IBM.2107-75ABTV1 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
dscli> lspprc 1000-1001 1100-1101
====================================================================================================
dscli>
<< DS8000 #2 >>

dscli> lspprc 2000-2001 2100-2101
===========================================================================================================
1000:2000 Target Suspended Update Target Global Copy 10 unknown Disabled Invalid
24.6.4 Perform Global Copy Failover from B to A

see 24.4.2, “Perform Global Copy Failover from B to A” on page 331. Give this command to
the DS HMC connected to the remote disk subsystem.
Example 24-46 Perform Global Copy Failover from B to A

<< DS8000 #2 >>
dscli> lspprc 2000-2001 2100-2101
===========================================================================================================
dscli>
dscli> failoverpprc -remotedev IBM.2107-7520781 -type gcp 2000-2001:1000-1001 2100-2101:1100-1101
dscli>

dscli> lspprc 2000-2001 2100-2101
====================================================================================================
dscli>
24.6.5 Create consistent data on B volumes

see 24.4.4, “Reverse FlashCopy from B to C” on page 334. Give the reverseflash command
to the DS HMC connected to the remote disk subsystem.
Example 24-47 Reverse FlashCopy B to C

dscli> reverseflash -fast -tgtpprc 2000-2001:2200-2201 2100-2101:2300-2301
24.6.6 Wait for the FlashCopy background copy to complete

After the FlashCopy background copy completes, the FlashCopy relationship ends. You can
check this with the lsflash command. See Example 24-48.
Example 24-48 Check the FlashCopy background copy completion

dscli> lsflash 2000-2001 2100-2101
CMUC00234I lsflash: No Flash Copy found.
24.6.7 Re-establish the FlashCopy relationships

In order to resume the Global Mirror environment quickly, we re-establish the FlashCopy
relationships from B to C with the original options for the Global Mirror environment; see
Example 24-49. For detailed discussion and considerations see 24.4.5, “Re-establish the
FlashCopy relationship from B to C” on page 337.
Give this command to the DS HMC connected to the remote disk subsystem.
Example 24-49 Reestablish the FlashCopy relationships

dscli> mkflash -tgtinhibit -nocp -record 2000-2001:2200-2201 2100-2101:2300-2301
dscli>
dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
24.6.8 Take FlashCopy from B to D

In the previous step we created a consistent copy of the data on the B volumes. Now we
make another copy of the B volumes for the disaster recovery testing. We call these
FlashCopy targets the D volumes. In our example we have four D volumes, which are 2400,
2401, 2500, and 2501; see Figure 24-14.

mkflash B to D
FlashCopy
LSS10 LSS20 LSS22 LSS24
1000 2000 2200 2400

Global Copy Pair
1001 2001 2201 2401
LSS11 FlashCopy
LSS21 LSS23 LSS26
1100 2100 2300 2500

Global Copy Pair
1101 2101 2301 2501
A B C D
GC: Source GC: Source FC: Target FC: Target
FC: Source
DS8000#1 DS8000#2
Figure 24-14 After mkflash B to D
Example 24-50 shows the DS CLI log for this operation. We use the -nocp option for the
FlashCopy. You can also use the copy option.
Example 24-50 Take FlashCopy B to D

dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
dscli>
dscli> mkflash -nocp 2000-2001:2400-2401 2100-2101:2500-2501
dscli>

dscli> lsflash 2000-2001 2100-2101
====================================================================================================================================
dscli>
24.6.9 Perform the disaster recovery testing using the D volume

Depending on your operating system and system environment, it may be necessary to
re-scan Fibre Channel devices and mount the D volume at the remote site. After this, you can
perform your disaster recovery testing using the D volumes. You can also use the D volumes
for backup or to make a tape backup.
24.6.10 Perform Global Copy Failback from A to B

In order to return to the normal Global Mirror environment, we have to resume the Global
Copy pairs we had suspended in a previous step. Because the application at the production
site keeps running and we must not lose these updates, we have to re-synchronize the A and
the B volumes — with the As being the source and the Bs being the target. Give this
command to the DS HMC connected to the local DS8000 (DS8000#1); see Figure 24-15.

failbackpprc A to B
FlashCopy
LSS10 LSS20 LSS22 LSS24
1000 2000 2200 2400

Global Copy Pair
1001 2001 2201 2401
LSS11 FlashCopy
LSS21 LSS23 LSS26
1100 2100 2300 2500

Global Copy Pair
1101 2101 2301 2501
A B C D
GC: Source GC: Target FC: Target FC: Target
FC: Source
DS8000#1 DS8000#2
Figure 24-15 Perform Global Copy Failback A to B - test scenario
For the failback operation we use the failbackpprc command; see Example 24-51.
Important: Do not specify the B volume as a source with the failbackpprc command (to
the DS8000#2), otherwise data on the B volume will be copied to the A volume. If the A
volume does not have reserve status, data on the A volume may be overwritten.
Example 24-51 Perform Global Copy Failback from A to B - test scenario

<< Before failbackpprc >>
<< DS8000 #1 >>
dscli> lspprc 1000-1001 1100-1101
====================================================================================================
dscli>
<< DS8000 #2 >>

dscli> lspprc 2000-2001 2100-2101
====================================================================================================
<< DS8000 #1 >>

dscli> failbackpprc -remotedev IBM.2107-75ABTV1 -type gcp 1000-1001:2000-2001 1100-1101:2100-2101
dscli>
dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>

24.6.11 Wait for the Global Copy first pass to complete
The Global Copy first pass needs to not complete to resume Global Mirror. However, the
Consistency Group formation does not start until this completion. You can check the status
with the lspprc command; see Example 24-52. The First Pass Status field indicates the
status of the first pass, where True means the first pass is complete. You can also use the
lssession command output field FirstPassComplete to verify the status of the first pass.
Example 24-52 Check the Global Copy first pass completion

dscli> lspprc 1000-1001 1100-1101
==================================================================================================
dscli>
dscli> lspprc -l 1000-1001 1100-1101
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs) Critical Mode First Pass
==============================================================================================================================================================
dscli>
=================================================================================================================
24.6.12 Resume Global Mirror

Now you can resume Global Mirror with the resumegmir command. You can verify the result
with the showgmir command; see Example 24-53. For detailed discussion and considerations
see 24.3.1, “Pause and resume Global Mirror Consistency Group formation” on page 320.
Give this command to the DS HMC connected to the local DS8000 (DS8000#1).
In our example, the first showgmir output shows Running in the Copy State field but the
Consistency Group formation is not done yet. You can find this from the Current Time and the
CG Time field. The next showgmir command shows that at least one Consistency Group
formation has been done after the showgmir command.
Example 24-53 Resume Global Mirror

dscli> resumegmir -dev IBM.2107-7520781 -lss IBM.2107-7520781/10 -session 02
dscli>
ID IBM.2107-7520781/10
Master Count 1
Copy State Running
Current Time 11/14/2005 21:31:15 JST
CG Time 11/14/2005 18:49:59 JST
FlashCopy Sequence Number 0x43785DC7
Subordinate Count 0
ID IBM.2107-7520781/10
Master Count 1
Copy State Running
Current Time 11/14/2005 21:31:30 JST
CG Time 11/14/2005 21:31:30 JST
FlashCopy Sequence Number 0x437883A2
Subordinate Count 0

In this section we explain how to create and manage a Global Mirror session using the DS
Storage Manager (DS SM) graphical user interface (GUI).
In the DS SM GUI examples we use two DS8000s, serial numbers 7503461 and 75ABTV1, in
a quite similar configuration as the one used in the previous examples. Still, in this case note
that 7503461 is at the local production site and 75ABTV1 is at the remote backup site. Also,
on each machine we use volumes in LSS 14. You can tell the volumes apart by their names.
The local volumes on serial 7503461 are named av_8K_1_1400 to av_8K_1_1403. The
remote volumes on serial 75ABTV1 are named av_8K_3_1400 to av_8K_3_1403.
We had Ethernet connectivity between the sites to connect to the storage complexes. We also
performed the procedure to add the remote storage complex to the local storage complex.
24.8 Setup a Global Mirror environment with the DS GUI

To set up a Global Mirror environment we follow a similar procedure as with the DS CLI. The
configuration we set up is discussed in 24.7, “DS Storage Manager GUI examples” on
page 355.
24.8.1 Define paths

With DS Storage Manager, to create new paths between two LSSs on two different storage
disk subsystems, it is necessary to go through a six-step process.

You need to repeat this wizard for each data path between LSSs on the source storage image
on site 1 and the target LSSs on the target storage image on site 2, and for each control path
between the master storage image LSS and the subordinate storage image LSSs.
To launch this wizard you need to go first to the Paths panel under the Copy Services menu of
the DS Storage Manager GUI; see Figure 24-16. Select from the pull-down lists the storage
complex, then the storage unit, then the storage image, and finally the LSS that contains the
source volumes of the Global Copy pairs we want to create. In the Select Action pull-down
list, choose Create to proceed with the first step of the wizard.
Figure 24-16 Global Copy paths creation - Launch the creation process
Now the creation wizard is displaying the Select source LSS panel; see Figure 24-17. Here
you select from the pull-down list, the LSS that contains the source volumes of the Global
Copy pairs you are going to create.
Figure 24-17 Global Copy paths creation - step 1: select the source LSS
Click Next to proceed with the second step of this wizard.
When the creation wizard displays the Select target LSS panel, select from the pull-down lists
the storage complex, then the storage unit, then the storage image, and finally the LSS that
contains the target volumes; see Figure 24-18.
Figure 24-18 Global Copy paths creation - step 2: select the target LSS
Click Next to proceed with the third step of this wizard.
When the creation wizard displays the Select source I/Os ports panel, select (using the check
boxes) at least one I/O port (two is better) to use for Global Copy replication; see
Figure 24-19. In the location column, four digits indicate the location of a port:
򐂰 The first digit (R) is to locate the frame.
򐂰 The second digit (E) is for the I/O enclosure.
򐂰 The third digit (C) is for the adapter.
򐂰 The fourth digit (P) is for the adapter’s port.
Figure 24-19 Global Copy paths creation - step 3: select the source I/O ports
Click Next to proceed with the fourth step of this wizard.

When the creation wizard displays the Select target I/Os ports panel, for each I/O port
selected during the third step, select from its related pull-down list the target I/O port; see
Figure 24-20. Refer to step 3 of this wizard for an explanation of the RECP digits.
In this example, no choices are being presented, as there is only one path for our port.
Figure 24-20 Global Copy paths creation - step 4: select the target I/O ports
When the creation wizard displays the Select path options panel, you can select using the
check box the option Define as Consistency Group; see Figure 24-21. This option is not
mandatory since the Global Mirror session will handle the consistency of the data across the
set of volumes.
Figure 24-21 Global Copy paths creation - step 5: select the paths options
Figure 24-22 shows the Verification panel. Here you check all the components of your paths
Figure 24-22 Global Copy paths creation - step 6: verification
24.8.2 Create Global Copy pairs

With DS Storage Manager, to create new Global Copy relationships for one or several volume
pairs, it is necessary to go through a five-step process.
Note: If Global Copy pairs are in several LSSs, do not forget to select all of them during
this wizard or you will have to run the wizard again on each LSS. If Global Copy pairs are
spread over several storage units, run this wizard again on each of them.
To launch this wizard, you need to go first to the Metro Mirror panel under the Copy Services
menu of the DS Storage Manager GUI; see Figure 24-23. In the Select Action pull-down list,
choose Create.
Figure 24-23 Global Copy creation - Launch the creation process

When the creation wizard displays the Volume Pairing Method, select the radio button
Manual volume pair assignment; see Figure 24-24.
Figure 24-24 Global Copy creation - step 1: choose volume pairing method
When the creation wizard displays the Select source volumes panel, select from the
pull-down lists the storage complex, the storage unit, the storage image, the resource type,
and if necessary its appropriate parameter to display the list of volumes. See Figure 24-25.
Figure 24-25 Global Copy creation - step 2: select the source volumes
If you have chosen the resource type LSS, then select from the pull-down list the LSS number
that contains the source volumes you are going to use for Global Mirror and then check the
boxes of the selected volumes.
Note: At this step, if paths have not yet been created we can click Create Paths to launch
the wizard described in 24.8.1, “Define paths” on page 355.
When the creation wizard displays the Select target volumes panel, we can notice that only
one source volume is indicated on the top of the panel; see Figure 24-26. This means that we
will repeat this third step for each volume that we have selected during the second step.
Figure 24-26 Global Copy creation - Step 3: select the target volume 1
Select from the pull-down lists the storage complex, then the storage unit, then the storage
image, then the resource type, and if it is necessary its appropriate parameter to display the
list of volumes. Then on the required LSS check the box for the selected target volume.
Click Next to proceed with the selection of the second target volume; see Figure 24-27.
Figure 24-27 Global Copy creation - Step 3: select the target volume 2

When the creation wizard once again displays the Select target volumes panel, we notice that
only the indicated source volume on the top of the panel is different. Once again, select from
the pull-down lists the storage complex, then the storage unit, then the storage image, then
the resource type, and if it is necessary its appropriate parameter to display the list of
volumes. Then on the required LSS check the box for the selected target volume.
If we had selected more source volumes, we would once again proceed to the next target
volume selection panel. Since it is the second and last volume in our selection, click Next to
proceed with the fourth step of this wizard.
When the creation wizard displays the Select copy options panel, select the radio button
Global Copy to define the type of replication, then check the box for Permit read access
from target, and if it is the first synchronization between source and target volumes, check
the box for Perform initial copy. See Figure 24-28.
Figure 24-28 Global Copy creation - Step 4: select the copy options
Click Next to proceed with the fifth and last step of this wizard.
In the Verification panel, verify all the components of your Global Copy session configuration
and, if necessary, click Back to correct any of them or click Finish to validate; see
Figure 24-29.
Figure 24-29 Global Copy creation - Step 5: verification
24.8.3 Create FlashCopy relationships

With the DS Storage Manager, to create new FlashCopy relationships for one or several pairs
of volumes, it is necessary to go through a five-step process.
Note: If FlashCopy pairs are in several LSSs, do not forget to select all of them during this
wizard or you will have to run the wizard again on each LSS. If FlashCopy pairs are spread
over several storage images run this wizard again on each of them.
Because they are for a Global Mirror environment, these FlashCopy pairs are created on the
remote machine using the Global Copy targets as the FlashCopy source. This means we
need to select the remote storage complex and the remote storage unit.

To launch this wizard you first need to go to the FlashCopy panel under the Copy Services
menu of the DS Storage Manager GUI. Select from the pull-down lists the storage complex,
then the storage unit, then the storage image, and finally the LSS that contains the Global
Copy target volumes. In the Select Action pull-down list choose Create to proceed with the
first step of the wizard; see Figure 24-30.
Figure 24-30 FlashCopy creation - Launch the creation process
When the creation wizard displays the Define relationship type, select the radio button A
single source with a single target. This is because we will have to activate the record option
on the fourth step of this wizard; see Figure 24-31.
Figure 24-31 FlashCopy creation - Step 1: Select the relationship type
When the creation wizard displays the Select source volumes panel, select from the
pull-down lists the storage complex, then the storage unit, then the storage image, then the
resource type, and if it is necessary its appropriate parameter to display the list of volumes. If
you have chosen the resource type LSS, select from the pull-down lists the LSS number that
contains the source volumes you want to use for the Global Mirror environment. Then check
the boxes of the selected source volumes. See Figure 24-32.
Figure 24-32 FlashCopy creation - Step 2: select the source volumes
When the creation wizard displays the Select target volumes panel, select from the pull-down
lists the resource type and, if it is necessary, its appropriate parameter to display the list of
volumes. If you have chosen the resource type LSS, select from the pull-down lists the LSS
number that contains the target volumes you want to use. Then check the boxes of the
selected target volumes. See Figure 24-33.
Figure 24-33 FlashCopy creation - Step 3: select the target volumes
Click Next to proceed with the fourth step of this wizard.

When the creation wizard displays the Select common options panel, check the box for
Enable change recording. This will automatically check the box for Make relationship(s)
persistent. You can leave the *Sequence number field with its default value since Global
Mirror will take care of it automatically. See Figure 24-34.
Note: Do not check the box for Initiate background copy. You may need to un-select it.
Figure 24-34 FlashCopy creation - Step 4: select the common option
Click Next to proceed with the fifth and last step of this wizard.
In the Verification panel, verify all the components of the FlashCopy configurations, and if
necessary click Back to correct any of them or click Finish to validate. See Figure 24-35.
Figure 24-35 FlashCopy creation - Step 5: verification
24.8.4 Create the Global Mirror session
With the DS Storage Manager, to create a Global Mirror session you go through a three-step
process.
To launch this wizard you need to go first to the Global Mirror panel under the Copy Services
menu of DS Storage Manager GUI; see Figure 24-36. Select from the pull-down lists the
storage complex, then the storage unit, and then the storage image that contains the source
volumes that will be part of the Global Mirror. Then from the Select Action pull-down list,
choose Create.
Figure 24-36 Global Mirror creation - Launch the creation process
The creation wizard displays the Select volumes panel; see Figure 24-37.
Figure 24-37 Global Mirror creation - Step 1: select volumes

When the creation wizard displays the Select volumes panel, click the required storage unit,
then the required LSS, and check the boxes for the source volumes you want to be part of the
Global Mirror session; see Figure 24-37 on page 367.
Note: At this step, if Global Copy pairs and FlashCopy pairs are not yet created, we can
click Create Metro Mirror to launch the wizard described in 24.8.2, “Create Global Copy
pairs” on page 359, and we can click Create FlashCopy to launch the wizard described in
24.8.3, “Create FlashCopy relationships” on page 363.
Figure 24-38 Global Mirror creation - Step 2: define properties
When the creation wizard displays the Define properties panel, the session number field
should be filled with the appropriate session number. We can click Get Available Session
Numbers if we have forgotten the one we want to use. See Figure 24-38.
There are also three important fields that can be set at this moment using this panel, and
these are the Global Mirror session tuning parameters. For a detailed understanding of these
options, see 20.4.2, “Consistency Group parameters” on page 261. Following is a brief
discussion of each of them:
򐂰 Consistency Group interval time (seconds): specifies how long to wait between the
formation of Consistency Groups. If this number is not specified or is set to zero,
Consistency Groups are formed continuously.
򐂰 Maximum coordination interval (milliseconds): indicates the maximum time that Global
Mirror can queue I/Os in the source disk subsystem to start forming a Consistency Group.
򐂰 Maximum time writes inhibited to remote site (seconds): specifies the maximum amount of
time allowed for the consistent set of data to drain to the remote site before failing the
current Consistency Group.
Click Next to proceed with the third and last step of this wizard.
In the Verification panel, check all the components of your Global Mirror session configuration
and, if necessary, click Back to correct any of them or click Finish to validate; see
Figure 24-39.
Figure 24-39 Global Mirror creation - Step 3 - Verification
To view the status of the Global Mirror session, go to the Global Mirror panel under the Copy
Services menu of the DS Storage Manager GUI. Select from the pull-down lists the storage
complex, then the storage unit, then the storage image, which is the master Global Mirror
session manager, and wait until the screen is refreshed. If it is necessary, click Refresh to
refresh the panel. See Figure 24-40.
Figure 24-40 Global Mirror - Visualize the session status
24.9 Manage the Global Mirror environment with the DS GUI

In this section we discuss and give examples of how to perform common Global Mirror control
tasks using the DS CLI. The following management activities are presented:
򐂰 Query a Global Mirror environment.
򐂰 Pause a Global Mirror session.
򐂰 Resume a Global Mirror session.
򐂰 Add or remove volumes from an LSS to the Global Mirror session.

Many of the tasks described in this section were already described in previous sections as
part of procedures like the site swap procedure or the disaster recovery test procedure.
24.9.1 View settings and error information of the Global Mirror session
To see session information you first to go to the Global Mirror panel under the Copy Services
menu of the DS Storage Manager GUI. Then check the box for the Global Mirror session for
which you want to display its properties, and in the Select Action pull-down list, choose
Properties; see Figure 24-41.
Figure 24-41 View Global Mirror session’s properties: Launch the viewing process
The Global Mirror session properties: Real-time panel will be displayed. In the General tab
you can review the setting for this session; see Figure 24-42.
Figure 24-42 View Global Mirror session settings: General tab
You can then click Failures to see the Failures tab; see Figure 24-43. In the Global Mirror
session properties: Real-time panel, on the Failures tab, you can request selected information
using the radio buttons Most recent failure, Previous failure, and First failure. Once you are
done reviewing the information, click OK to finish and go back to the main Global Mirror
panel.
Figure 24-43 View Global Mirror session errors information - Failures tab
24.9.2 View information of volumes in the Global Mirror session

To launch this viewing action you need to first go to the Global Mirror panel under the Copy
Services menu of the DS Storage Manager GUI. Check the box for the Global Mirror session
for which you want to display its associated volumes information, and in the Select Action
pull-down list, choose View session volumes to proceed; see Figure 24-41 on page 370.
The Global Mirror session volumes: Real-time panel will be displayed and it will show
information about the volumes associated with the Global Mirror session; see Figure 24-44.
You can either download or print this information table. Then click OK to finish and go back to
the main Global Mirror panel.
Figure 24-44 Global Mirror session volumes information

24.9.3 Pause a Global Mirror session
To pause a Global Mirror session you first go to the Global Mirror panel under the Copy
Services menu of the DS Storage Manager GUI. Then select the Global Mirror session with
which you want to work. Then in the Select Action pull-down list, choose Pause to proceed
with the first step of the wizard; see Figure 24-41 on page 370.
When the Global Mirror: Real-time panel displays the warning message (see Figure 24-45)
either click Cancel to return to the main Global Mirror panel without pausing the Global Mirror
session or OK to pause the Global Mirror session and to return to the main Global Mirror
panel.
Figure 24-45 Pause Global Mirror: Confirm the pause of the Global Mirror session
When the main Global Mirror panel is displayed, note that the state of the Global Mirror
session is now Paused; see Figure 24-46.
Figure 24-46 Pause Global Mirror: View session status paused
24.9.4 Resume a Global Mirror session

To resume a Global Mirror session first go to the Global Mirror panel under the Copy Services
menu of the DS Storage Manager GUI; see Figure 24-41 on page 370. Then select the
Global Mirror session that you are going to resume. Then in the Select Action pull-down list
choose Resume to proceed.
When the Global Mirror Real-time panel displays the warning message (see Figure 24-47),
either click Cancel to return to the main Global Mirror panel without resuming the Global
Mirror session or click OK to resume the Global Mirror session and to return to the main
Global Mirror panel. When the main Global Mirror panel is displayed, the state of the Global
Mirror session will now show Running.
Figure 24-47 Resume Global Mirror - Confirm the resume of the Global Mirror session
24.9.5 Modify a Global Mirror session

When using the DS Storage Manager to modify an existing Global Mirror session, either to
add or remove one or several sets of volumes or to change the Global Mirror settings, it is
necessary to go through a three-step process.
To launch the wizard you first go to the Global Mirror panel under the Copy Services menu of
the DS Storage Manager GUI. Then select the Global Mirror session you wish to work with.
Then in the Select Action pull-down list, choose Modify to proceed with the first step of the
wizard; see Figure 24-41 on page 370.
When the modification wizard displays the Select volumes panel, click the required storage
unit, then the required LSS, and either select or un-select the check box for the source
volumes you wish to either add or remove from the Global Mirror session. The panel will look
similar to Figure 24-37 on page 367.
Note: At this step, if Global Copy pairs and FlashCopy pairs have not been created yet,
you can click Create Metro Mirror to launch the wizard described in 24.8.2, “Create Global
Copy pairs” on page 359, and you can click Create FlashCopy to launch the wizard
described in 24.8.3, “Create FlashCopy relationships” on page 363.
When the creation wizard displays the Define properties panel, the session number field
should be filled with the appropriate session number. This panel will look similar to
Figure 24-38 on page 368. In this panel there are three important fields you can modify at this
moment: Consistency Group interval time (seconds), Maximum coordination interval
(milliseconds), and Maximum time writes inhibited to remote site (seconds). For a detailed
discussion of these Global Mirror tuning parameters see 20.4.2, “Consistency Group
parameters” on page 261.
Click Next to proceed with the third and last step of this wizard. This is the Verification panel.
Here you verify all the components of your Global Mirror session configuration and, if
necessary, click Back to correct any of them or click Finish to validate.

Part 7
Part 7 Metro/Global
Mirror
This part gives an understanding of a new function for the DS8000, Metro/Global Mirror, a
3-site disaster recovery solution.

25
Chapter 25. Metro/Global Mirror overview

This chapter describes the characteristics and operation of IBM System Storage Metro/Global
Mirror, a three-site, high availability, disaster recovery implementation. Also discussed are the
considerations for its implementation on the IBM System Storage DS8000.
Metro/Global Mirror is not supported on DS6000.

25.1 Metro/Global Mirror overview
Metro/Global Mirror is a three-site, multi-purpose, replication solution for both System z and
open systems data. As shown in Figure 25-1, Metro Mirror provides high availability
replication from local site (site A) to intermediate site (site B), while Global Mirror provides
long distance disaster recovery replication from intermediate site (site B) to remote site
(site C).
Server or Servers
***
normal application
I/Os Global Mirror network Global Mirror FlashCopy
asynchronous incremental
Metro Mirror long distance NOCOPY
A B C
D
Metro Mirror
network Global Mirror
synchronous
short distance
Local Site (site A) Intermediate Site (site B) Remote Site (site C)
Figure 25-1 Metro/Global Mirror elements
Both Metro Mirror and Global Mirror are well established replication solutions. Metro/Global
Mirror combines Metro Mirror and Global Mirror to incorporate the best features of the two
solutions:
򐂰 Metro Mirror
– Synchronous operation supports zero data loss.
– The opportunity to locate the intermediate site disk subsystems close to the local site
disk subsystems allows use of intermediate site disk subsystems in a high availability
configuration.
Note: Metro Mirror is supported to a distance of up to 300 km but, when in a

Metro/Global Mirror implementation, a shorter distance might be more appropriate in
support of the high availability functionality.
򐂰 Global Mirror
– Asynchronous operation supports long distance replication for disaster recovery.
– Global Mirror methodology has no impact on applications at the local site.
– Provides a recoverable, restartable, consistent image at the remote site with a
Recovery Point Objective (RPO) typically in the 3–5 second range.
This chapter provides a high level overview of Metro/Global Mirror. The details of the
individual processes/elements of the solution, for example, Metro Mirror and Global Mirror,
will not be repeated here as they are already described in great detail in respective parts of
this book.
25.1.1 Metro/Global Mirror design objectives

The development of the Metro/Global Mirror solution was based upon the following
requirements and principles:
򐂰 System z data or open systems data or both
Metro Mirror and Global Mirror are replication technologies that support both System z
data and open systems data separately or intermixed on the same disk subsystems.
򐂰 Proven technology
Both Metro Mirror and Global mirror are technologies that have existed in client production
environments for an extended period.
򐂰 Consistent, recoverable data
– Metro Mirror - synchronous copy and Freeze/Run support this requirement.
– Global Mirror - momentarily pausing the primary application write I/Os for
approximately 3 msec every 3–5 seconds. This allows formation of consistent groups
of updates that are drained to the remote site and FlashCopied to create the
appropriate image.
򐂰 High availability - mirrored data immediately available and accessible by local hosts
The Metro Mirror image is a zero data loss image that can be placed in or near the local
site and can be quickly made available to the local site applications.
򐂰 Disaster recovery - mirrored data at long distance for regional disasters
Global Mirror is asynchronous and the remote site can be at continental distance.
򐂰 Recovery Point Objective (RPO) in single digit seconds
– Metro Mirror - RPO = zero
– Global Mirror - RPO = typically 3-5 seconds
򐂰 Scalable solution - accommodate growth, across many disk subsystems
– Metro Mirror - There are no architectural limits on the number of disk subsystems.
– Global Mirror - There are some current limits on the total number disk subsystems.
25.2 Metro/Global Mirror processes

Figure 25-2 on page 380 illustrates the overview of Metro/Global Mirror processes. The
Metro/Global Mirror process is understood if the component processes are understood. There
is one notable difference in that the intermediate site volumes (site B volumes) are special
because they act as both source (GM) and target (MM) volumes at the same time.
The local site (site A) to intermediate site (site B) component is identical to Metro Mirror.
Application writes are synchronously copied to the intermediate site before write complete is
signaled to the application. All writes to the local site volumes in the mirror are treated in
exactly the same way. This is explained in great detail in Chapter 11, “Metro Mirror overview”
on page 109, of this book.
Chapter 25. Metro/Global Mirror overview 379

The intermediate site (site B) to remote site (site C) component is identical to Global Mirror,
except that:
򐂰 The writes to intermediate site volumes are Metro Mirror secondary writes and not
application primary writes.
򐂰 The intermediate site volumes are both source (GM) and target (MM) at the same time.
The intermediate site disk subsystems are collectively paused by the Global Mirror master
disk subsystem to create the Consistency Group (CG) set of updates. This pause would
normally take 3 ms every 3 to 5 seconds. After the CG set is formed, the Metro Mirror writes,
from local site (site A) volumes to intermediate site (site B) volumes, are allowed to continue.
Also, the CG updates continue to drain to remote site (site C) volumes. The intermediate site
to remote site drain is expected to take only a few seconds to complete, perhaps as few as 2
or 3 seconds.
Once all updates are drained to the remote site, all changes since the last FlashCopy from
the C volumes to the D volumes are logically (NOCOPY) FlashCopied to the D volumes. After
the logical FlashCopy is complete, the intermediate site to remote site Global Copy data
transfer is resumed until the next formation of a Global Mirror Consistency Group. The
process described above is repeated every 3 to 5 seconds if the interval for Consistency
Group formation is set to zero. Otherwise it will be repeated at the specified interval plus 3 to
5 seconds.
The Global Mirror processes are presented in much greater detail in Chapter 20, “Global
Mirror overview” on page 243.
Server or Servers
***
4
normal application I/Os
Global Mirror network Global Mirror FlashCopy
asynchronous incremental
Metro Mirror long distance NOCOPY
1
2
A B C
3 b
a c
D
Metro Mirror network
synchronous Global Mirror
short distance
Metro Mirror write Global Mirror consistency group formation (CG)

1. application to VolA a. write updates to B volumes paused (< 3ms) to create CG
2. VolA to VolB b. CG updates to B volumes drained to C volumes
3. write complete to A c. after all updates drained, FlashCopy changed data from C to D
4. write complete to application volumes
Local Site (Site A) Intermediate Site (Site B) Remote Site (Site C)
Figure 25-2 Metro/Global Mirror overview diagram
26
Chapter 26. Configuration and setup

This chapter gives an overview of possible setups of the Metro/Global Mirror and how it could
be made applicable for an open systems environment. It also contains a hands-on section on
how to set up a simple Metro/Global Mirror environment.

26.1 Metro/Global Mirror configuration
As described in Chapter 25, “Metro/Global Mirror overview” on page 377, the Metro/Global
Mirror is a cascade of a Metro Mirror and a Global Mirror. The Metro Mirror covers the data
replication between local site and intermediate site. The Global Mirror is cascaded from the
secondary volumes of the Metro Mirror and continues to copy the data from intermediate site
to remote site.
26.1.1 Metro/Global Mirror with additional Global Mirror

A recovery of the storage at the remote site could be initiated for various reasons, such as a
disaster situation at the local site or a planned failover, to avoid the risk of production impact
in case of maintenance operations at the local site. After a recovery at the remote site and
when production has been started at the remote site, it is likely that this status will persist for
an extended period of time. During that time, the data at the remote site is not protected
against a disaster situation that may occur at the remote site.
One method to be prepared for a disaster at the remote site is to use the storage at the
intermediate site for a target copy. Because the bandwidth between the remote and the
intermediate site will generally be too low to set up a Metro Mirror, the only possible way to
copy the data is to establish a Global Copy relationship. Global Copy by itself does not
guarantee that the copied data is consistent, but a Global Mirror relationship can provide the
necessary data consistency. To set up a Global Mirror between the remote and intermediate
sites, an additional set of FlashCopy volumes is required at the intermediate site. Figure 26-1
illustrates the volume setup for Metro/Global Mirror with an additional Global Mirror.
Local Site Intermediate Site Remote Site
Global Mirror
E from remote
to intermediate site
A B C
Metro Mirror
Global Mirror
D
from intermediate
to remote site
Figure 26-1 Metro/Global Mirror with additional Global Mirror from remote to intermediate site
26.1.2 Metro/Global Mirror with multiple storage subsystems
Based on the capabilities of Global Mirror, it is possible to span the Metro/Global Mirror
across multiple storage subsystems. At the intermediate site one of the storage subsystems
has the role of the Global Mirror master and the other ones are the subordinates. The Global
Mirror master controls the formation of the Consistency Group. It requires additional RMC
(PPRC) links between the master and each subordinate. See Figure 26-2.
The Metro Mirror will be set up from multiple storage subsystems at the local site to the
intermediate to the Global Mirror primary systems. The paths of the Metro Mirror relations
must be established with consistency group enabled.
Local site Intermediate site Remote site
Global Mirror
Master
Metro Mirror
Global Mirror
Global Mirror
Metro Mirror
Subordinate
Global Mirror
Metro Mirror
Subordinate
Figure 26-2 Setup of a Metro/Global Mirror with multiple storage subsystems
26.2 Configuration examples

In an open systems environment, large applications are typically distributed across multiple
servers according to their components and functionality. For example, database-related
applications typically consist of a database engine and a certain number of application
Chapter 26. Configuration and setup 383

servers acting as front ends. In larger configurations you may find additional servers that offer
file services. All these components may be part of a single application.
In a disaster situation where a failover to the intermediate or remote site has to be done, all
servers and storage that are required to start up an application are subject to the takeover.
Thus a disaster recovery concept must be worked out for each application individually. It
defines which components and functionality must be recovered at the intermediate or remote
sites in case of a disaster.
In Figure 26-3 a possible configuration of a Metro/Global Mirror for an open systems

environment is illustrated. It is divided into a primary production, a secondary production, and
a remote site. Between the primary and secondary production sites Metro Mirror relations can
be configured in either direction. In both production sites a Global Mirror is configured to the
remote site. All connections are set up with redundant inter-switch links between fiber
directors or switches.
Note: The schematic in Figure 26-3 shows only one Fibre Channel director per site. In a
real implementation the connections between the sites should be realized by two
redundant fabrics across all locations.
Primary Secondary
production site production site Remote site
Global Mirror
DS8000 DS8000 DS8000
Metro Mirror
Global Mirror
Metro Mirror
2109-M14
2109-M14
Primary 2109-M14 Secondary

single Host single Host
P590
P590
P550 P550 P550 P550 P590
HACMP Remote
stretched clusters production for
stretched clusters
Figure 26-3 Configuration example for an open systems environment
When an application uses a Metro Mirror from the primary production site to the secondary
production site, the primary production site is the Metro/Global Mirror local site and the
secondary site is the intermediate site and vice versa for applications running in the other
direction.
This setup allows you to configure cluster systems that span across both production sites,
offering more flexibility for high availability setups. A failure of a server at the production site
can be taken over automatically at the other production site using the automatic take over
feature of the cluster software while the primary storage is still being used. Otherwise, if a
single host system, as shown in Figure 26-3 on page 384, fails at the primary production site,
the only possibility to bring up the production again is a recovery of the storage and the server
at the remote site.
A failure of the storage can be seen as a failure of an infrastructure component, and this can
be categorized as at least a partial disaster. In this case, a recovery of storage at the
intermediate or remote site has to be performed. The storage at the intermediate site can be
accessed by the server at the local site, when the bandwidth between both production sites is
high enough. This implies that there is not necessarily a takeover of the servers required,
which offers more flexibility on how to start up the applications.
For cost effectiveness it is possible to consolidate the needed disk capacity at the remote site.
Because of the distance, a stretched cluster environment may not be possible, so single host
systems or local clustered systems are implemented at the remote site. A large scale server
which is capable of providing multiple logical partitions to run the multiple applications from
the production sites could be used. It is also possible to equip the storage subsystems at the
remote site with disk drive modules of higher capacity to reduce the number of installed
storage subsystems.
26.3 Initial setup of Metro/Global Mirror

In this section we describe how to set up a Metro/Global Mirror environment. For each step
we provide sample commands and their output using the DS CLI.
Figure 26-4 shows the logical setup of the Metro/Global Mirror. It also shows the steps in the
order of the operational sequence. Considering the potentially large amount of data that has
to be copied, we recommend that you first set up Global Copy with the NOCOPY option.
When the Metro Mirror is created and starts to synchronize from the local site to the
intermediate site, the Global Copy passes the tracks to the remote site.
To avoid performance impact on a running production, we recommend that you start the
Metro Mirror initial copy when the production write activity is low.
Local Intermediate Remote

Site Site Site
A 1 3 B 1 2 C
4
6
D
5
Figure 26-4 Set up Metro/Global Mirror

As shown in Figure 26-4 on page 385, the setup of the Metro/Global Mirror is done in the
following steps:
1. Set up all Metro Mirror and Global Mirror paths.
2. Set up Global Copy with NOCOPY from intermediate site to remote site.
3. Set up Metro Mirror between local and intermediate sites.
4. Set up FlashCopy at the remote site.
5. Create a Global Mirror session and add volumes to the session at the intermediate site.
6. Start Global Mirror at the intermediate site.
26.3.1 Identifying the PPRC ports

For performance reasons, it is very important that the links between sites are used exclusively
for PPRC paths. In general, it is possible to mix ports for PPRC paths with host connections,
but because of the high link utilization for PPRC communication, it may impact host I/Os.
To identify the ports to which the PPRC links are connected you can use the lsavailpprcport
command. This command displays all local and remote ports which have a physical
connection. This could be either a point-to-point link or a zoned connection through a storage
area network.
In Example 26-1 you can see that there are four PPRC links (physical paths) available. These
connections can be used to create the PPRC paths according to the user’s installation plan.
Example 26-1 Display available PPRC ports
dscli> lsavailpprcport -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 -remotewwnn

5005076303FFCE63 -fullid 70:72
Date/Time: June 19, 2006 2:48:43 PM CEST IBM DSCLI Version: 5.1.600.196 DS:
IBM.2107-75ABTV1
Local Port Attached Port Type
==================================================
IBM.2107-75ABTV1/I0033 IBM.2107-75ABTV2/I0233 FCP
IBM.2107-75ABTV1/I0102 IBM.2107-75ABTV2/I0301 FCP
26.3.2 Step 1: Set up all Metro Mirror and Global Mirror paths
Based on the PPRC links that are discovered, the paths have to be created for each LSS pair
in the Metro/Global Mirror configuration. There are two prerequisites which have to be fulfilled
to create PPRC paths successfully:
򐂰 The physical connection must be available, either by an appropriate zone in a storage
area network or by point-to-point links.
򐂰 The LSSs must exist in both storage devices. This means that at least one volume of the
designated LSSs must exist.
It is a good practice to create the PPRC paths for both directions. In case of a disaster all
paths in all directions are available and do not need to be created in the critical situation of the
disaster.
Example 26-2 shows the setup of Metro Mirror paths between the local and intermediate
sites. The mkpprcpath command is always executed at the DS8000 where the source
volumes are for each direction. This means that for the direction local to intermediate, the
mkpprcpath is executed at the local site and vice versa for the opposite direction.
Example 26-2 Set up PPRC paths between local and intermediate site
At local site:
dscli> mkpprcpath -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 -remotewwnn

5005076303FFCE63 -srclss 70 -tgtlss 72 -consistgrp I0033:I0233 I0102:I0301
IBM.2107-75ABTV1
At intermediate site:

5005076303FFC663 -srclss 72 -tgtlss 70 -consistgrp I0233:I0033 I0301:I0102
IBM.2107-75ABTV2
Note: To ensure consistency of the Metro Mirror, the PPRC paths have to be created with
the option -consistgrp. Between the intermediate site and the remote site, the Global Mirror
is responsible for the creation of the Consistency Group. This means that the paths for the
Global Copy must not be set up with the -consistgrp option.
Example 26-3 shows the mkpprcpath command for the Global Mirror. Again, the paths are
created for both directions and the commands are executed at the DS8000 where the source
is.
Example 26-3 Set up PPRC paths between intermediate site and remote site
dscli> mkpprcpath -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75BYGT1 -remotewwnn

5005076304FFC671 -srclss 72 -tgtlss 74 I0230:I0231 I0303:I0301
IBM.2107-75ABTV2
At remote site:
dscli> mkpprcpath -dev IBM.2107-75BYGT1 -remotedev IBM.2107-75ABTV2 -remotewwnn

5005076303FFCE63 -srclss 74 -tgtlss 72 I0231:I0230 I0301:I0303
IBM.2107-75BYGT1
CMUC00149I mkpprcpath: Remote Mirror and Copy path 74:72 successfully
established.

26.3.3 Step 2: Set up Global Copy NOCOPY from intermediate to remote sites
The Global Copy relationship will be cascaded from a Metro Mirror relationship. To enable the
cascade, the option -cascade has to be supplied with the mkpprc command. In Example 26-4
the complete command is shown. In step 3 the Metro Mirror will be created and will copy all
data from local site volumes to intermediate site volumes. The data are then forwarded by the
Global Copy to remote site volumes. To avoid copying the data twice, the initial setup of the
Global Copy is initiated with NOCOPY mode.
Example 26-4 Set up Global Copy from intermediate to remote sites

dscli> mkpprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75BYGT1 -type gcp -mode nocp -cascade
7200-7203:7400-7403
Date/Time: June 19, 2006 3:57:00 PM CEST IBM DSCLI Version: 5.1.600.196 DS: IBM.2107-75ABTV2
26.3.4 Step 3: Set up Metro Mirror between local and intermediate sites
To set up the Metro Mirror the mkpprc command is used, where the option -type mmir signifies
that a Metro Mirror will be created. For the initial setup the pairs are created in the full copy
mode. Example 26-5 shows the mkpprc command, which is executed at the local site.
Example 26-5 Set up Metro Mirror from the local site to the intermediate sites
dscli> mkpprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 -type mmir -mode full
7000-7003:7200-7203
dscli>
The volumes at the intermediate site are target volumes for Metro Mirror and source volumes
for Global Copy at the same time. When the lspprc command is executed against these
volumes, it shows the pair status of both the Metro Mirror and Global Copy relationships, as
illustrated in Example 26-6.
Example 26-6 Query the PPRC relationship at the intermediate site

dscli> lspprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75BYGT1 -fullid -fmt default
7200-7203
IBM.2107-75ABTV2
ID State Reason Type
===========================================================================================
==========================================================
IBM.2107-75ABTV1/7000:IBM.2107-75ABTV2/7200 Target Full Duplex - Metro Mirror
IBM.2107-75ABTV1/70 unknown Disabled Invalid
IBM.2107-75ABTV2/7200:IBM.2107-75BYGT1/7400 Copy Pending - Global Copy
IBM.2107-75ABTV2/72 unknown Disabled True
IBM.2107-75ABTV2/72 unknown Disabled
True
26.3.5 Step 4: Set up FlashCopy at remote site

When the Metro Mirror relations have reached Full Duplex state, set up the FlashCopy at the
remote site. It requires the following options:
-tgtinhibit The -tgtinhibit option prevents host writes to the FlashCopy target
volumes as long as the FlashCopy relationship exists. This protects
FlashCopy target volumes when the volumes are assigned a host by
accident.
-record With the -record option the changed tracks of the FlashCopy
relationship will be recorded at both the source and target volumes.
This is used by the re-synchronize of the FlashCopy, which is
periodically triggered by the Global Mirror.
-persist The -persist options retains the FlashCopy relationship. A persistent
FlashCopy is required to enable the recovery of the consistency in
case of the failover to the remote site.
-nocp The -nocp option will prevent a background copy of the Global Copy
target volumes. The purpose of the FlashCopy at the remote site is
just to save the changed tracks since the FlashCopy has been set up.
Example 26-7 shows how to create the FlashCopy for the Global Mirror at the remote site.
Example 26-7 Create the FlashCopy for the Global Mirror at the remote site
dscli> mkflash -dev IBM.2107-75BYGT1 -tgtinhibit -record -persist -nocp 7400-7403:7600-7603
IBM.2107-75BYGT1
26.3.6 Step 5: Create Global Mirror session and add volumes to session
This is done at the intermediate site. The session is created with the mksession command.
For each LSS a session has to be created. The session is denoted by a session number. Only
one session can be active in a storage subsystem. This means that all LSSs in a Global
Mirror must have the same session number.

Example 26-8 shows that in a first step an empty session is created. The second command
populates the session with the primary volumes of the Global Copy. As long as the Global
Mirror has not been started and no consistency group has been formed, the status of the
volumes will be Join Pending.
Example 26-8 Create the sessions and add the volumes

dscli> mksession -dev IBM.2107-75ABTV2 -lss 72 1
dscli>
dscli> chsession -dev IBM.2107-75ABTV2 -lss 72 -action add -volume 7200-7203 1
dscli>
dscli> lssession -dev IBM.2107-75ABTV2 72
LSS ID Session Status Volume VolumeStatus PrimaryStatus SecondaryStatus FirstPassComplete
AllowCascading
======================================================================================================
===============
72 01 Normal 7200 Join Pending Primary Copy Pending Secondary Full Duplex True
Enable
Enable
Enable
Enable
26.3.7 Step 6: Start Global Mirror at intermediate site

Finally the Global Mirror is started. The mkgmir command requires the session number and
one of the LSSs which are configured in the session. In Example 26-9 the lssession
command shows that the volumes are now in the active state.
Example 26-9 Start the Global Mirror

dscli> mkgmir -dev IBM.2107-75ABTV2 -lss 72 -session 1
IBM.2107-75ABTV2
IBM.2107-75ABTV2
LSS ID Session Status Volume VolumeStatus PrimaryStatus SecondaryStatus
FirstPassComplete AllowCascading
===========================================================================================
==================================
72 01 CG In Progress 7200 Active Primary Copy Pending Secondary Full
Duplex True Enable
Duplex True Enable
Duplex True Enable
Duplex True Enable
With the command showgmir it can be verified if the Global Mirror was successfully created.
The Copy State should show Running. See Example 26-10.
Example 26-10 Monitor Global Mirror

dscli> showgmir -dev IBM.2107-75ABTV2 72
IBM.2107-75ABTV2
ID IBM.2107-75ABTV2/72
Master Count 1
Copy State Running
Current Time 06/19/2006 16:18:55 CEST
CG Time 06/19/2006 16:18:55 CEST
FlashCopy Sequence Number 0x4496B24F
Master ID IBM.2107-75ABTV2
Subordinate Count 0
When the option -metrics is supplied with the showgmir command the progress of the
Consistency Group formation can be monitored. The entry Total Successful CG Count shows
the current number of successful created Consistency Groups. When the Global Mirror is
running the number of Consistency Groups is steadily growing each time the showgmir
command is issued. See Example 26-11.
Example 26-11 Show progress of consistency formation

dscli> showgmir -dev IBM.2107-75ABTV2 -metrics 72
IBM.2107-75ABTV2
Last Successful CG Form Time 06/19/2006 16:19:55 CEST
First Failure LSS -
Last Failure LSS -
dscli> showgmir -dev IBM.2107-75ABTV2 -metrics 72

IBM.2107-75ABTV2
Last Successful CG Form Time 06/19/2006 16:19:58 CEST
First Failure LSS -
Last Failure LSS -
The command showgmiroos displays the number of tracks which are out of synchronization.
Example 26-12 shows the OutOfSyncTracks of LSS 72.
Example 26-12 Display the number of out-of-sync tracks

dscli> showgmiroos -dev IBM.2107-75ABTV2 -lss 72 -scope lss 1
IBM.2107-75ABTV2
Scope IBM.2107-75ABTV2/72
Session 01
OutOfSyncTracks 0
The status of the session is displayed with the command lssession. It displays the session
status of all volumes of an LSS. The command can take a list of LSSs to inspect the session
status of volumes from multiple LSSs.
26.4 Going from Metro Mirror to Metro/Global Mirror

Many installations today use Metro Mirror to provide high availability replication across two
sites. This setup can be extended to provide long-distance disaster recovery replication to a
third site using Metro/Global Mirror.
In this section we describe the procedure on how to migrate from a two-site Metro Mirror
setup (local site to intermediate site) to a three-site Metro/Global Mirror setup (local site to
intermediate site to remote site) using DS CLI. This can be done while production is up and
running at the local site. Depending on the capability of the link between the intermediate site
and the remote site, the initial copy of the data may be more or less influenced by write
activity of the applications. If possible, the initial copy should be started during a low write
activity of the application. But in any case, the first pass of the Global Copy must be
completed for the Global Mirror to be able to form consistency groups.
Note: To ensure consistency from the local to the remote site it is essential that the Metro
Mirror paths are created with consistency enabled. If this was not the case for the existing
Metro Mirror, the paths for each LSS can be changed using the chlss command.
A B 1 2 C
3
4
D
5
Figure 26-5 Migrating from Metro Mirror to Metro/Global Mirror
As shown in Figure 26-5, the steps required to extend an existing Metro Mirror to a three-site
setup using Metro/Global Mirror are:
1. Set up PPRC paths from the intermediate site to the remote site.
2. Set up Global Copy with COPY from the intermediate site to the remote site.
In contrast to the Metro/Global Mirror setup, the Global Copy is established with copy
mode to ensure that all data is copied from the intermediate site to the remote site.
3. Set up FlashCopy at the remote site.
4. Create a Global Mirror session and add volumes to the session at the intermediate site.
The steps are described in detail in 26.3, “Initial setup of Metro/Global Mirror” on page 385.
26.5 Recommendations for setting up Metro/Global Mirror

This section gives some brief recommendations for the planning of the setup of Metro/Global
Mirror.
򐂰 Use LSS to group volumes logically to applications.
Because some Copy Services functions manage LSSs, for ease of management we
recommend that you dedicate LSSs to one application. For example, a freezepprc
command freezes all volumes of the specified LSS. To manage applications
independently, each application should use dedicated LSSs.
򐂰 Use at least two PPRC links.
We strongly recommend that you implement at least two independent PPRC links each for
Metro Mirror and Global Mirror for redundancy. Capacity planning for Metro Mirror may

yield a requirement for more than two links, because the synchronous relationship
requires a higher bandwidth. With Global Copy only the last modified version of a track is
transmitted to the secondary volumes, so the bandwidth requirement is not as high as it is
for Metro Mirror. However, to provide redundancy at least two independent links should be
deployed for the Global Copy.
򐂰 Use at least two LSSs per host (one per server node).
For performance we recommende that you use at least two LSSs per host that are
processed by both server nodes of the DS8000. The volume identifiers that are assigned
at volume creation time determine this distribution. Even-number LSSs are processed by
server 1 and odd-number LSSs are processed by server 2.
򐂰 Volume placement for FlashCopy at the remote site.
FlashCopy is a copy process that copies data within the storage subsystem. To ensure
that this happens in the most efficient way it is essential that the copy paths inside the
storage subsystem be as short as possible. For this reason FlashCopy target volumes
should use LSSs that are processed by the same server as the FlashCopy source
volumes. Otherwise the data has to be passed to the other server.
27
Chapter 27. General Metro/Global Mirror

operations
In this chapter we discuss general considerations for Metro Mirror and Global Mirror when
used within the context of Metro/Global Mirror. We also give hints and tips related to the
specific operation of Metro/Global Mirror.

27.1 Definitions
For reference, we define some terms used in this chapter:
Host A host is a server where applications or components of
applications are running. Hosts may be implemented as single
servers or may be clustered with other servers.
Applications An application comprises all software components that are used to
build a self-contained solution for the user’s business. The different
software components are running on one or more servers.
Application takeover Applications that are running on cluster servers can take
advantage of the take-over procedures offered by the cluster
software. In a failure situation the cluster software stops the
application from where it is currently running and starts it
automatically at the other clustered server.
Primary storage In a remote copy relation the primary storage is where the regular
production resides; it is the source for the data replication.
Secondary storage In a remote copy relation the secondary storage is where the data
is replicated to. A secondary storage is the target of the copy
relation.
Storage failover A storage failover changes the access point of the data from the
primary to the secondary storage subsystem. The application is
started using the secondary storage subsystem.
27.2 General consideration for storage failover

A failover of the storage has a serious impact on a running production. It results in down time
for the applications, because they must be started with the storage of the secondary site.
Depending on the configuration of the storage environment, access to the secondary storage
has to be configured after the failover and before the applications can be started up.
Thus a storage failover is only activated for situations such as:

򐂰 Resulting from a disaster and where required infrastructure components are now
inaccessible. See Chapter 30, “Unplanned scenarios” on page 435.
򐂰 In case of maintenance operations at the local site.
򐂰 Migration of applications.
Disaster recovery tests can be performed without failover of the applications.
27.3 Check pair status before failover

Before doing a failover of the Metro Mirror or the Global Copy relationship, it is essential that
the status of the pair relations is checked for synchronicity. For a planned failover the status
of the Metro Mirror at the primary and secondary sites must be in the Full Duplex mode. If the
status is still Copy Pending, it means that the data is not completely copied to the secondary
volumes.
In Example 27-1 all volumes of the Metro Mirror are in Full Duplex mode.
Example 27-1 Full Duplex Mode of a Metro Mirror

dscli>lspprc -dev IBM.2107-7520781 -remotedev IBM.2107-75ABTV1 -fullid -fmt default
6000-6003
ID State Reason Type SourceLSS
Timeout (secs) Critical Mode First Pass Status
===========================================================================================
===================================================
IBM.2107-7520781/6000:IBM.2107-75ABTV1/6200 Full Duplex - Metro Mirror
IBM.2107-7520781/60 unknown Disabled Invalid
The Global Copy volume status will always be Copy Pending. To identify that all tracks were
copied to the secondary site the Out of Sync Tracks must be obtained. The Global Copy is
synchronized when for all volume pairs the out Out of Sync Tracks has the value zero, as
Example 27-2 Synchronized Global Copy relation

dscli> lspprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 -l -fullid -fmt default
6400-6403
ID State Reason Type Out Of Sync
Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs)
===========================================================================================
===========================================================================================
===========================
IBM.2107-75ABTV2/6400:IBM.2107-75ABTV1/6200 Copy Pending - Global Copy 0
Disabled Enabled invalid - IBM.2107-75ABTV2/64 unknown Disabled
True
True
True
True
In a Metro/Global Mirror the volumes at the intermediate site are in a different state than in a
conventional Metro Mirror or Global Mirror. Because the Global Mirror is cascaded to the
Metro Mirror, the volumes at the intermediate site are target and source at the same time.
Thus the lspprc command issued at the intermediate site will show both the Metro Mirror and
the Global Mirror, as shown in Example 27-3.
Example 27-3 Status of the intermediate volumes in a Metro/Global Mirror

dscli> lspprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-7520781 -fullid -fmt default
6200-6203
Chapter 27. General Metro/Global Mirror operations 397

===========================================================================================
==========================================================
IBM.2107-7520781/6000:IBM.2107-75ABTV1/6200 Target Full Duplex - Metro Mirror
IBM.2107-75ABTV1/6200:IBM.2107-75ABTV2/6400 Copy Pending - Global Copy
27.4 Freeze and unfreeze Metro Mirror volumes

The freezepprc and unfreezepprc commands are working in a Metro Mirror when the PPRC
paths are created with the -consistencygrp option. A freezepprc is doing two different things:
򐂰 It sets the long busy condition on the primary volumes of the given LSS. This causes the
I/O to these volumes to be blocked. All I/O from the hosts will wait until the long busy
condition is removed by the unfreezepprc command or when the long busy timeout has
been exceeded. The default timeout value is 120 seconds. Since it is a value regarding
the LSS, this value can be changed by the chlss command using the option -extlongbusy
timeout.
򐂰 The paths between the primary site and the secondary site are removed.
For a detailed description of how consistency is provided in a Metro Mirror see 12.4,
“Consistency Group function” on page 116.
As shown in Example 27-4, the pair status at the primary site after the freezepprc is now
Suspended with the reason Freeze. At the secondary site the pairs status is still in Target Full
Duplex mode.
Example 27-4 Example of a freezepprc and the pair status at the primary site
At the local DS8000:
dscli> freezepprc -dev IBM.2107-7503461 -remotedev IBM.2107-75ABTV1 68:62

dscli> lspprc -dev IBM.2107-7503461 -remotedev IBM.2107-75ABTV1 -fmt default 6800-6803
Status
===========================================================================================
=====
At the intermediate DS8000:
dscli> lspprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-7503461 -fmt default 6200-6203
ID State Reason Type SourceLSS Timeout (secs) Critical Mode
First Pass Status
===========================================================================================
==============
6800:6200 Target Full Duplex - Metro Mirror 68 unknown Disabled
Invalid
Invalid
Invalid
Invalid
The unfreezepprc removes the long busy status from the primary volumes and I/O will
continue. The pair status of the primary volumes is still Suspended, as shown in Example 27-5
Example 27-5 Unfreezepprc after the freezepprc

dscli> unfreezepprc -dev IBM.2107-7503461 -remotedev IBM.2107-75ABTV1 68:62
dscli> lspprc -dev IBM.2107-7503461 -remotedev IBM.2107-75ABTV1 -fmt default 6800-6803
Status
===========================================================================================
=====
A freezepprc command can be issued against a running application, although it has an

impact on the application: after the freezepprc the applications are waiting to continue with
the I/O. This is either enabled by the unfreezepprc or after a timeout of 120 seconds. This
method is used to provide a consistent copy of the data at the secondary site when the
applications at the primary site are not stopped.
Important: We recommend that you run both commands in one script to ensure that the
time delay between the freeze and the unfreeze is as small as possible.
27.5 Suspend volumes before failover

In a Metro Mirror that is in full duplex mode, a failoverpprc command will be issued against
the secondary volumes. This will change the status of the secondary volumes from Target
Full Duplex to Host Suspended, while the primary volumes remain in Full Duplex mode. This
has two implications:
򐂰 If host I/O is still ongoing, the primary storage subsystem tries to copy the data to the
secondary volumes. Because they are suspended after the failover, this I/O fails since it
finds the secondary volume in a wrong state. The primary volumes then go into the
suspend mode as well. This failure causes an error message to be sent to the host. The
host then sends a retry and I/O will continue.
򐂰 If the Metro Mirror will be re-established from the primary volumes to the secondary
volumes, the failbackpprc will fail because it requires that the primary volumes are in
suspend state. The primary volumes must be put in suspend state with a pausepprc
command with the option -unconditional issued at the primary site.

A good practice is to suspend the Metro Mirror or Global Copy before any failover operation.
27.6 Remove volumes from the session

The Global Mirror forms consistency groups for all volumes in the active Global Mirror
session. This requires that the Global Copy volumes are in the Copy Pending state and that
the FlashCopy relation exists. If one or more volumes in an active session are suspended or
even removed, the Global Mirror is no longer able to form consistency groups because one or
more members of the session are in the wrong state.
Assuming an environment as described in in 26.2, “Configuration examples” on page 383,

multiple applications may be managed by the Global Mirror. In planned scenarios only one or
a few applications, but not all applications, will be processed for a failover. It is important that
the Global Mirror is able to continue forming Consistency Groups for those applications that
will reside in production at the local site.
To achieve this, the failed-over volumes have to be removed from the Global Mirror session.
This removes these volumes from the focus of the Global Mirror, and the other volumes are
processed by the Global Mirror. To ensure that the volumes that have to be removed from the
session are consistent, the Global Mirror should be paused until all volumes are removed.
After the Global Mirror is resumed, Consistency Groups will be formed for the remaining
volumes and the failover procedure can continue. Example 27-6 shows the commands.
Example 27-6 Removing volumes from the Global Mirror session

dscli> pausegmir -dev IBM.2107-75ABTV1 -session 1 -lss 54
Date/Time: December 11, 2005 4:02:20 PM CET IBM DSCLI Version: 5.1.0.204 DS:
IBM.2107-75ABTV1
dscli>
dscli> chsession -dev IBM.2107-75ABTV1 -action remove -volume 54d4-54d7 -lss 54 1
IBM.2107-75ABTV1
dscli>
dscli> resumegmir -dev IBM.2107-75ABTV1 -session 1 -lss 54
IBM.2107-75ABTV1
27.7 Check consistency at the remote site

The Metro Mirror from the local to the intermediate site ensures consistency using the freeze
and unfreeze functionality. The paths must be created with the -consistgrp option. The
consistency is provided to the Metro Mirror by an extended long busy condition, which is set to
the primary volumes, when all the Metro Mirror links have failed. When I/O to the primary
volumes can resume, an unfreezepprc command is issued to remove the extended long busy
condition.
The Global Mirror is a process running at the intermediate site, which uses the FlashCopy at
the remote site. When more than one storage subsystem is used at the intermediate site, one
of these is denoted as the master where the Global Mirror resides (see 26.1.2, “Metro/Global
Mirror with multiple storage subsystems” on page 383). The other storage subsystems are
the subordinates. The Global Mirror coordinates the consistency formation for all volumes of
the master and the subordinates, which are joined into the Global Mirror session. The
consistency is formed in three steps:
1. The master is coordinating with all subordinates to stop the I/O for 3–5 msec to all primary
volumes to form a Consistency Group.
2. The Consistency Group is transmitted to the secondary volumes.
3. The FlashCopy copies all tracks that have been changed since the last write operation to
the secondary volumes.
See 20.4, “Consistency Groups” on page 259, for a detailed description of cosistency
formation in a Global Mirror.
Create consistency group by holding

application writes while creating bitmap Transmit updates in Global Copy mode while
containing updates for this consistency between consistency groups
group on all volumes - design point is Consistency group interval - 0s to 18hrs
2-3ms. FlashCopy issued
Maximum coordination time eg 10ms with revertible option
Drain consistency group and send to FlashCopy committed Start next consistency group
remote DS8000 using Global Copy once all revertible
Application writes for next flashcopies have
consistency group are recorded in successfully completed
change recording bitmap
Maximum drain time - eg 1 min
Figure 27-1 Phases of consistency group formation
The consistency of the data can be provided in each phase of the consistency group
formation process, as shown in Figure 27-1:
򐂰 When the failure occurs during the coordination time or the draining of the data to the
remote site, consistency is still available on the FlashCopy volumes, because the
FlashCopy has not yet started.
򐂰 When the failure happens while the FlashCopy command is executed, a manual
intervention is required to either roll revert or commit the consistency group before
continuing with the recovery or restarting Global Mirror. The action that is indicated
depends on the current status of the FlashCopy, where the sequence numbers and the
revertible flag are of special interest:
– When the sequence numbers of the FlashCopy are different, the copy process has not
started for all the volumes. In this case the recent FlashCopy is inconsistent and can
not be used and must be rolled back with the revertflash command, which removes
all not-committed sequences from the FlashCopy target.
– When the sequence numbers are all equal and there is a mix of revertible and
unrevertible volumes, the copy to the FlashCopy targets has taken place but the
process has not been finished for some volumes. In this case the resent FlashCopy
targets are usable and the process has to be committed manually by a commitflash.

Example 27-7 shows a commit situation. All volumes shown in the lsflash command have
the same sequence number. The volumes 50E8 and further have the Revertible flag enabled
while the volumes above the flag are disabled. In this case the commitflash command must
be issued to the volumes 50E8 and further to recreate the consistency.
Example 27-7 Commit situation

dscli> lsflash -dev IBM.2107-75ABTV2 -l -fmt default 5000-50ff 5100-51ff 5300-5344
ID SrcLSS SequenceNum Timeout ActiveCopy Recording Persistent Revertible
SourceWriteEnabled TargetWriteEnabled BackgroundCopy OutOfSyncTracks DateCreated
DateSynced
===========================================================================================
===========================================================================================
========================
5000:5A00 50 437DABED 300 Disabled Enabled Enabled Disabled Enabled
Disabled Disabled 15259 Fri Nov 18 09:22:09 CET 2005 Fri Nov 18
11:24:44 CET 2005
5001:5A01 50 437DABED 300 Disabled Enabled Enabled Disabled Enabled
11:24:44 CET 2005
.....
50E7:5AE7 50 437DABED 300 Disabled Enabled Enabled Disabled Enabled
11:24:44 CET 2005
50E8:5AE8 50 437DABED 300 Disabled Enabled Enabled Enabled Disabled
11:24:44 CET 2005
50E9:5AE9 50 437DABED 300 Disabled Enabled Enabled Enabled Disabled
11:24:44 CET 2005
.....
Example 27-8 shows a revertible situation. The sequence numbers have two different values,
whereby for the lower sequence number the revertible flag is disabled. This shows that for
these volumes the FlashCopy process has not yet started. The volumes with the higher
sequence numbers have the revertible flag enabled, which means that the relationship exists
but is not committed. The correct interaction to bring back consistency is to issue a
revertflash command.
Example 27-8 Revertible situation

dscli> lsflash -dev IBM.2107-75ABTV2 -l -fmt default 5000-50ff 5100-51ff 5300-5344
SourceWriteEnabled TargetWriteEnabled BackgroundCopy OutOfSyncTracks DateCreated
DateSynced
===========================================================================================
===========================================================================================
========================
5000:5A00 50 437DC7BD 300 Disabled Enabled Enabled Enabled Disabled
13:23:24 CET 2005
5001:5A01 50 437DC7BD 300 Disabled Enabled Enabled Enabled Disabled
13:23:24 CET 2005
.....
50E3:5AE3 50 437DC7BD 300 Disabled Enabled Enabled Enabled Disabled
13:23:24 CET 2005
50E4:5AE4 50 437DC7BC 300 Disabled Enabled Enabled Disabled Enabled
13:23:23 CET 2005
13:23:24 CET 2005
13:23:23 CET 2005
13:23:24 CET 2005
13:23:23 CET 2005
13:23:23 CET 2005
50EA:5AEA 50 437DC7BD 300 Disabled Enabled Enabled Enabled Disabled
13:23:24 CET 2005
50EB:5AEB 50 437DC7BD 300 Disabled Enabled Enabled Enabled Disabled
13:23:24 CET 2005
When the option -revertible is supplied with the lsflash command, only the revertible
volumes are listed.
As a final step to provide complete and consistent data, after a failover to a remote site, the
data located at the FlashCopy target volumes must be reversed to the source volumes. After
the failover to the Global Mirror secondary site, the host access is gained to the secondary
volumes, which are also the FlashCopy source volumes.
After a failover to the remote site the host access is gained to the secondary volumes of the
Global Copy relation. But some last-saved consistent data may be located at the FlashCopy
target volumes. In this case the FlashCopy target volumes must be reversed to their sources,
which are also the secondary Global Copy volumes.
In Example 27-9 the command of the fast reverse restore is shown.
Note: When the application has been stopped and two consistency groups have been
formed, we can assume that the data on the FlashCopy source and on the FlashCopy
target are the same. In this case a fast reverse restore is not necessary.
Example 27-9 Fast reverse restore of the FlashCopy target volumes

dscli> reverseflash -dev IBM.2107-75ABTV2 -fast 6400-6403:6600-6603
27.8 Set up an additional Global Mirror from remote site

When a planned failover of the production to the remote site has been fulfilled, depending on
the reason for the failover, it may be that the production will remain for an extended period of

time at the remote site. If the intermediate site is still available it is possible to set up an
additional Global Mirror from the remote site to the intermediate site to protect the data
against a possible disaster at the remote site.
To do so requires that additional volumes are available to set up a FlashCopy relation with the
volumes at the intermediate site as the FlashCopy source volumes. After the failover to the
remote site is completed (see 28.4, “Recovery at remote site” on page 416, for the details) a
failback of the Global Copy will be used to copy the data to the intermediate site. Finally, a
session with the related volumes must be created and the Global Mirror must be started at
the remote site.
Figure 27-2 shows the steps to set up the additional Global Mirror.
E
2
A B C
3
Figure 27-2 Set up an additional Global Mirror after failover to remote site
The setup of the additional Global Mirror comprises the following steps:
1. Create or fail back from the remote site to the intermediate site.
2. Establish FlashCopy to the additional volumes at the intermediate site.
3. Create a session and start up the Global Mirror.
27.8.1 Step 1: Create Global Copy from remote to intermediate site

The volumes at C (see Figure 27-2) are now a valid source for the Global Mirror, which is
established from the remote site to the intermediate site. The first step is to set up the Global
Copy from the remote volumes to the intermediate volumes.
Example 27-10 shows how to set up the Global Copy. Because the data at the remote site is
the same as at the intermediate site, the Global Copy can be established with the option
-mode nocp to avoid background copy. The -cascade option has to be omitted because the
Global Copy is now not a cascaded relation.
Example 27-10 Setup Global Copy from intermediate to remote site
At the remote site:
dscli> mkpprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 -type gcp -mode nocp
6400-6403:6200-6203
xecuting command:
created.
created.
created.
created.
dscli>
dscli> lspprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 -l -fullid -fmt default
6400-6403
ID State Reason Type Out Of Sync
Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs)
===========================================================================================
===========================================================================================
===========================
True
True
True
True
27.8.2 Step 2: Create FlashCopy at the intermediate site

The Global Mirror at the remote site requires a FlashCopy relation at the secondary site from
the volumes at B to a new set of volumes named E (see Figure 27-2 on page 404).

Example 27-11 shows how to set up the FlashCopy.
Example 27-11 Set up FlashCopy at intermediate site

dscli> mkflash -dev IBM.2107-75ABTV1 -tgtinhibit -record -persist -nocp
6200-6203:6300-6303
27.8.3 Step 3: Create session and Global Mirror at remote site

To complete the setup of the Global Mirror a session has to be created at the remote site and
the Global Copy source volumes have to be added into the session. Finally, the Global Mirror
is started.
Example 27-12 shows the setup of the session and the Global Mirror and how to check if the
Global Mirror is running properly.
Example 27-12 Create session and start up Global Mirror

dscli> mksession -dev IBM.2107-75ABTV2 -lss 64 1
dscli> lssession 64
===========================================================================================
==============================
64 01 CG In Progress 6400 Active Primary Copy Pending Secondary Simplex
True Enable
True Enable
True Enable
True Enable
dscli> mkgmir -dev IBM.2107-75ABTV2 -lss 64 -session 1
dscli> showgmir 64
Master Count 1
Copy State Running
Current Time 11/14/2005 10:38:21 CET
CG Time 11/14/2005 10:38:21 CET
FlashCopy Sequence Number 0x43785B0D
Master ID IBM.2107-75ABTV2
Subordinate Count 0
28
Chapter 28. Planned recovery scenarios

This chapter describes planned recovery scenarios. For each scenario, we describe in detail
all operations in a step-by-step approach. This chapter can be used as a cookbook for
Metro/Global Mirror operations.

28.1 Overview
Planned recovery scenarios are a series of operations initiated by the user and are based on
failover/failback advanced copy function features. A storage failover always implies an impact
to the production. For this reason all failover operations need to be planned very carefully by
the user in terms of integrity of procedures, time schedule, and availability of the applications.
In a planned failover the application has to be stopped before recovery at the intermediate
site. The host is then given access to the volumes at the intermediate site and the application
can be started using these volumes. A reason for a planned recovery at the intermediate site
could be because of maintenance activities at the local site that impact production (see 27.2,
“General consideration for storage failover” on page 396). A recovery at the intermediate site
minimizes the impact to the production normally running at the local site.
In a large data center many applications running on different servers may participate to the
Metro/Global Mirror. In this case it is most likely that failover operations will not be
accomplished against the complete environment but rather to specific applications. In this
case the failover operations have to be applied in a way that will not affect the copy relations
for the remaining applications. This situation is accounted for in all the scenarios presented in
this chapter.
Also, all the secnarios presented here have been practically tested and represent the best
practice for the situations they address. However, additional or alternate scenarios remain
possible depending on particular circumstances within your data center.
Note: If other scenarios are required, we strongly recommend that you extensively test
them before they are exercised in the production environment.
28.2 Recovery at intermediate site

A recovery at the intermediate site is indicated when the applications can be started on
servers that are located at the intermediate site. The advantage of a recovery at the
intermediate site is that the re-synchronization between the local and the intermediate site is
quite fast (during the failback function), because the bandwidth for the Metro Mirror is usually
higher than the bandwidth between the intermediate and the remote site.
When the servers have a connectivity from the local site to the storage at the intermediate
site it is also possible to start the applications at the local site. This makes sense when the
maintenance operations at the local site only affect the storage system, but not the servers. In
this case the recovery process for the application becomes simpler.
A recovery at the intermediate site is in fact a simple failover feature of the Metro Mirror. The
Global Mirror remains untouched and continues to copy data to the remote site.
2 3
A B C
1
4 D
Local production Prodution

host host at
intermediate site
Figure 28-1 Recovery of production to the intermediate site
The steps to failover the production to the intermediate site are as follows:
1. Stop the production application at the local site.
2. Suspend the Metro Mirror.
3. Failover the Metro Mirror to the intermediate site.
4. Start the application at the intermediate site.
28.2.1 Step 1: Stop the production application at the local site

When the production application has to move to the intermediate site the application I/O
needs to be stopped. It will be restarted after failover of the volumes at the intermediate site.
To prepare for the failback at a later point in time, the primary volumes must be released by
the host operating system. For the AIX operating system, all volume groups containing
volumes that belong to the Metro Mirror must be varied off.
28.2.2 Step 2: Suspend the Metro Mirror

Suspending the Metro Mirror is not mandatory, but is recommended prior to any failover
operations. See 27.5, “Suspend volumes before failover” on page 399, for details. The
failoverpprc command will change the status of the target (secondary) volumes but not the
source (primary) volumes. A suspend right before the failover will set both the primary and the
secondary volumes into the suspend state.
Example 28-1 illustrates how to bring the Metro Mirror pair into the suspended state. Before
the failoverpprc command is issued, the status of the volume pairs at the local and the
intermediate sites will be checked.
Example 28-1 Suspend the Metro Mirror

At the local site:
dscli> pausepprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 7000-7003:7200-7203
Chapter 28. Planned recovery scenarios 409

IBM.2107-75ABTV1
CMUC00157I pausepprc: Remote Mirror and Copy volume pair 7000:7200 relationship
successfully paused.
dscli> lspprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 -fullid 7000-7003
IBM.2107-75ABTV1
===========================================================================================
======================================================
IBM.2107-75ABTV1/7000:IBM.2107-75ABTV2/7200 Suspended Host Source Metro Mirror
IBM.2107-75ABTV1/70 300 Disabled Invalid
At the intermediate site:
dscli> lspprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75BYGT1 -fullid 7200-7203

IBM.2107-75ABTV2
===========================================================================================
===============================================================
IBM.2107-75ABTV1/7000:IBM.2107-75ABTV2/7200 Target Suspended Update Target Metro Mirror
28.2.3 Step 3: Failover the intermediate site

After the Metro Mirror has been suspended a failover to the secondary volumes can be
issued. When the failoverpprc command has successfully completed, the volumes at the
intermediate site can be accessed by the host.
Note: To failover to the intermediate site you have to specify the secondary volumes as the
source and the primary volumes as the target in the failoverpprc command.
Example 28-2. shows how to failover to the intermediate site using the failoverpprc
command.
Example 28-2 Failoverpprc at the intermediate site

dscli> failoverpprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 -type mmir
7200-7203:7000-7003
IBM.2107-75ABTV2
dscli> lspprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75BYGT1 -fullid 7200-7203
IBM.2107-75ABTV2
===========================================================================================
=============================================================
IBM.2107-75ABTV1/7000:IBM.2107-75ABTV2/7200 Target Suspended Host Target Metro Mirror
Note that after a failover, the status of the Metro Mirror secondary volumes used in a
Metro/Global Mirror is different than the status of the failover in a conventional Metro Mirror. In
a conventional Metro Mirror the secondary volumes will be in the status Suspended with
reason Host Source. Because in a Metro/Global Mirror the Metro Mirror secondary volumes
are still cascaded to the remote volumes, the status of these volumes will not be Suspended
Host Source. In the current implementation the status of the secondary volumes is displayed
as Target Suspended with the reason Host Target.
28.2.4 Step 4: Start the production application at the intermediate site

After you have varied on the volume groups and mounted the filesystems in case of an AIX
operating system, the applications can be started.

28.3 Return to local from intermediate site
This section describes how to move the production back to the local site after a transition to
the intermediate site. It relates to the scenario described in 28.2, “Recovery at intermediate
site” on page 408, or when the production has been taken over to the intermediate site after
failure at the local site.
The return from the intermediate site to the local site requires that the Global Copy must be
suspended. This is because the volumes cannot be the source for the remote and the local at
the same time. This enables two possibilities for the applications at the intermediate site that
have to be stopped:
1. When the data at the local site is too old because the production was kept for a relatively
long time at the intermediate site, it is safer to rely on the data in the Global Mirror. The
Global Mirror was always active, and consistent data is most current at the remote site. In
case of a problem during the failback, the data can be recovered from the Global Mirror.
For the failback process the applications should be stopped before the failback is issued.
The application can be started up at the local site, when the volume status of the Metro
Mirror is Full Duplex.
2. If the application downtime must be as short as possible, the applications may be stopped
after the failback to the local site, when all volumes are in Full Duplex. But this requires to
terminate the Global Copy. During the time when the intermediate volumes are
synchronized with the local volumes, the applications have no valid copy.
In Figure 28-2 the steps to move back to local is illustrated, whereby the application is
stopped first, to ensure that there is always a valid copy of the data available.

6 7
3 8
A B C
5 4
9 2 1 D
10
Disaster
Production
Recovery
Host
Test
Host
Figure 28-2 Return from intermediate site to local site
The steps are:

1. Stop I/O at the intermediate site.
2. Terminate Global Mirror or remove volumes from the Global Mirror session.
3. Suspend Global Copy.
4. Fail back Metro Mirror to the local site and wait for Full Duplex volume status.
5. Suspend Metro Mirror.
6. Fail over to the local site.
7. Fail back Metro Mirror from the local site to the intermediate site.
8. Resume Global Copy.
9. Start I/O at the local site.
10.Start Global Mirror or add volumes to the session.
28.3.1 Step 1: Stop I/O at the intermediate site

The Global Copy has to be suspended to enable the failback from the intermediate volumes
to the local volumes. In order to keep a valid and consistent copy of the data at the remote
site the applications running at the intermediate site need to be stopped right now. The
volumes must also be released by the host to enable the failback.
28.3.2 Step 2: Terminate Global Mirror or remove volumes from the session
Typically Metro/Global Mirror is deployed in an environment with more than one host
connected to the Metro/Global Mirror storage. Terminate Global Mirror session in case all
hosts are failed over to the intermediate site. If all hosts are not failed over to the intermediate
site, during the failback to the local site, the Global Mirror should be kept up and running to
ensure that the data for those hosts which remain at the local site is still copied in a consistent
way.
Prior to the failback the volumes belonging to the host which will failback, have to be removed
from the session. When this is done the Global Copy for these volumes can be suspended.
See Example 28-3.
Example 28-3 Remove the volumes from the session and pause the Global Copy
dscli> chsession -dev IBM.2107-75ABTV2 -lss 72 -action remove -volume 7200-7203 1
IBM.2107-75ABTV2 CMUC00147I chsession: Session 1 successfully modified.
28.3.3 Step 3: Suspend Global Copy

Before the failback of the Metro Mirror from the intermediate site to the local site can be
accomplished the status of the volumes at the intermediate needs to be Suspend Host Source.
This implies that the Global Copy between the intermediate and the remote site must be
suspended.
Example 28-4 shows how to suspend the Global Copy.
Example 28-4 Suspend Global Copy

dscli> pausepprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75BYGT1 7200-7203:7400-7403

28.3.4 Step 4: Fail back Metro Mirror to local site and wait for Full Duplex
Now the failback to the local site can be started. The failbackpprc command is executed at
the intermediate site, as shown in Example 28-5. Note that the secondary volumes are
specified as the source volumes and the primary volumes as the target volumes. Also, check
that the Metro Mirror volumes are in the Full Duplex mode before proceeding to the next step.
Example 28-5 Fail back to the local site and check the status
dscli> failbackpprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 -type mmir
7200-7203:7000-7003
IBM.2107-75ABTV2
dscli> lspprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 -fullid -fmt default
7200-7203
IBM.2107-75ABTV2
===========================================================================================
===================================================
IBM.2107-75ABTV2/7200:IBM.2107-75ABTV1/7000 Full Duplex - Metro Mirror
28.3.5 Steps 5 and 6: Suspend Metro Mirror and fail over to the local site
When all the volumes at the local site are synchronized with the intermediate site, the Metro
Mirror must be failed over to the local site before the applications can be started.
In Example 28-6 prior to the failover the Metro Mirror is suspended by a pausepprc command
according to the recommendation in 27.5, “Suspend volumes before failover” on page 399. At
this point the I/O to the volumes has been stopped and the direction of the mirror will be
changed with the next failback.
Note: To failover to the intermediate site, you have to specify the secondary volumes as
the source and the primary volumes as targets in the failoverpprc command.
Example 28-6 Suspend and failover to the local site

IBM.2107-75ABTV2
7000-7003:7200-7203
IBM.2107-75ABTV1
28.3.6 Step 7: Fail back Metro Mirror from the local site to the intermediate site
The failback from the local to the intermediate site triggers the Metro Mirror to copy the tracks
that are out-of-sync immediately after I/O to the primary volumes has been started.
Example 28-7 shows the failback to the intermediate site. This command is executed at the
local site.
Example 28-7 Fail back from the local to the intermediate site
7000-7003:7200-7203
IBM.2107-75ABTV1
28.3.7 Step 8: Resume Global Copy

Because the Global Copy was suspended (see 28.3.3, “Step 3: Suspend Global Copy” on
page 413) the Global Copy must now be resumed with the resumepprc command, which is
Example 28-8 Resume the Global Copy

dscli> resumepprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75BYGT1 -type gcp -cascade
7200-7203:7400-7403
IBM.2107-75ABTV2
CMUC00158I resumepprc: Remote Mirror and Copy volume pair 7200:7400 relationship
28.3.8 Step 9: Start I/O at the local site

Now when the Metro Mirror has been reversed, the applications can be started at the local
site. The Metro Mirror is ready to copy tracks. Since the Global Copy is still not up and the
volumes need to be added to the session, we recommend that the remaining steps are

processed promptly to avoid having Global Copy catch too many tracks when production is
running. This also ensures that Consistency Groups are formed as soon as possible.
28.3.9 Step 10: Start Global Mirror or add volumes to the session
In order to form Consistency Groups, start the Global Mirror session if it was terminated (see
28.3.2, “Step 2: Terminate Global Mirror or remove volumes from the session” on page 413)
or add the volumes to the session again if the Global Mirror is still up and running (as would
be the case when other applications that are not the subject of the failover/failback scenarios
are present and and need to have the Global Mirror kept in operation).
The volumes are added to the session with the chsession command using the option -action
add, as shown in Example 28-9. When done, the volume status will be Join Pending as long
as the first pass copy of the Global Copy has not been finished. Use the lssession command
to query the staus of the volumes in an LSS.
Example 28-9 Add volumes to the session and check the session
IBM.2107-75ABTV2
dscli> lssession -dev IBM.2107-75ABTV2 -fmt default 72
IBM.2107-75ABTV2
===========================================================================================
==================================
Duplex True Enable
Duplex True Enable
Duplex True Enable
Duplex True Enable
28.4 Recovery at remote site

As described in 28.2, “Recovery at intermediate site” on page 408, one possible reason for a
recovery of the production is to minimize impact to the production at the local site, for
example, in case of maintenance at the local site. The recovery at the remote site is indicated
when no servers are available at the intermediate site.
Figure 28-3 illustrates the steps it takes to recover to the remote site. This scenario includes
the setup of a Global Mirror from the remote site to the intermediate site in order to be
prepared for keeping the production at the remote site for an extended period of time.
Therefore the data will be replicated to the intermediate site.
4 5 6
A B C
3
1 2 7 D
Secondary
Primary production
production host
host
Figure 28-3 Recovery of production to remote site
When the application will remain for an extended period of time at the remote site and the
intermediate is still available, it is possible to create an additional Global Mirror from the
remote site to the intermediate site in order to protect the data against a possible disaster at
the remote site. See 27.8, “Set up an additional Global Mirror from remote site” on page 403,
for the details.
The recovery of the production to the remote site comprises the following steps:
1. Stop I/O at the local site.
2. Terminate Global Mirror or remove volumes from the session.
3. Terminate Global Copy.
5. Fail over Metro Mirror to the intermediate site.
6. Establish Global Copy from the remote site to the intermediate site.
7. Start I/O at the remote site.
28.4.1 Step 1: Stop I/O at the local site

Before any failover can happen the I/O to the primary volumes must the stopped. For this
purpose, applications running on hosts that are subject of the failover should be stopped
immediately. Data must be identical for the local, intermediate, and remote sites. During the
whole scenario make sure that data is not changing at the local site. The volumes must also
be released by the host to enable the failbackpprc of the Metro Mirror when the production
will return again to the local site (see 28.5.1, “Step 1: Stop I/O at remote site” on page 422).

28.4.2 Step 2: Terminate Global Mirror or remove volumes from session
As described in 28.2, “Recovery at intermediate site” on page 408, terminate Global Mirror if
all applications at the local site should be recovered at the remote site. If not all applications
from the local site will be transferred to the remote site and to ensure that Consistency
Groups continue to be formed for the applications that reside at the local site, the volumes by
the transferred applications will be removed from the session of the Global Mirror.
Example 28-10 shows how to remove volumes from the sessions.
Example 28-10 Remove volumes from the session

dscli> pausegmir -dev IBM.2107-75ABTV2 -session 1 -lss 72
Date/Time: June 21, 2006 10:34:41 AM CEST IBM DSCLI Version: 5.1.600.196 DS:
IBM.2107-75ABTV2
dscli> chsession -dev IBM.2107-75ABTV2 -lss 72 -action remove -volume 7200-7203 1
IBM.2107-75ABTV2
dscli> resumegmir -dev IBM.2107-75ABTV2 -session 1 -lss 72
IBM.2107-75ABTV2
28.4.3 Step 3: Terminate Global Copy

To prepare for te return back to the local site and in case the intermediate site is still available,
the Global Copy will be reversed from the remote site to the intermediate site. Because of the
cascading status of the intermediate volumes, a reverse of the Global Copy using the failover
and failback would result in the situation where the intermediate volumes are target volumes
for the Metro Mirror and the Global Copy at the same time. Since this is not suported, the
Global Copy must be terminated in this step. Later in this scenario (see 28.4.6, “Step 6:
Establish Global Copy from remote to intermediate site” on page 420) the Global Copy is
recreated in no-copy mode between the remote site and the intermediate site.
Attention: Between the current step (step 3) and up through step 6, make sure that all I/O
to the volumes are suspended. If not, data may be corrupted due to the no copy option,
which would then require a full copy.
Example 28-11 shows how to remove the Global Copy.
Important: Before the Global Copy is removed, check that all tracks were copied to the
secondary site. Use the command lspprc -l for obtaining the out-of-sync tracks.
Example 28-11 Remove Global Copy

dscli> rmpprc -quiet -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75BYGT1 7200-7203:7400-7403
IBM.2107-75ABTV2
CMUC00155I rmpprc: Remote Mirror and Copy volume pair 7200:7400 relationship successfully
withdrawn.
withdrawn.
withdrawn.
withdrawn.
28.4.4 Step 4: Suspend Metro Mirror

It is not necessary to suspend the Metro Mirror before the failover, because when production
will be transmitted to the local site, the failback will change the status of the secondary
volumes to Host Source Suspended. The status of the primary will remain unchanged.
However, to be prepared for the failback that will be issued later, it is a good practice to
suspend the Metro Mirror to bring the status of both the primary and the secondary volumes
into the suspended status before the failover.
Important: Before the Metro Mirror is suspended, check that all tracks were copied to the
intermediate site. Use the lspprc -l command for obtaining the out-of-sync tracks.
Note: If the links between the local and the intermediate site have failed, suspending the
Metro Mirror is not applicable, because the pair status has been turned into suspended.
Example 28-1 on page 409 illustrates how to bring the Metro Mirror pair into the suspended
state. Before the failoverpprc command is issued, the status of the volume pairs at the local
and the intermediate site must be checked.
Example 28-12 Suspend the Metro Mirror

At local site:

IBM.2107-75ABTV1
28.4.5 Step 5: Failover Metro Mirror to intermediate site

The volumes at the intermediate site will become secondary volumes for the Global Copy,
which will subsequently be set up between the remote and intermediate sites. Therefore the
Metro Mirror must be failed over to the intermediate site.
The Metro Mirror failover can also be seen as a preparation for the failback to the local site.
The reversed Metro Mirror will act as a cascade from the reversed Global Copy between the
remote and intermediate sites. For this reason the Metro Mirror secondary volumes failover is
issued with the -cascade option. Also, to avoid extended response times at the host side, a
cascaded copy relation should not be set up as a synchronous relation. Thus, the
failoverpprc command for the Metro Mirror will be executed with the option -type gcp.

Example 28-13 shows the failover of the Metro Mirror.
Note: To fail over to the intermediate site you must specify the secondary volumes as the
source and the primary volumes as targets in the failoverpprc command.
Example 28-13 Failover of the Metro Mirror

dscli> failoverpprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 -type gcp -cascade

7200-7203:7000-7003
IBM.2107-75ABTV2
28.4.6 Step 6: Establish Global Copy from remote to intermediate site

In anticipation of returning the production back to the local site and if the volumes at the
intermediate and the links are still available, it is now possible to establish the Global Copy
from the remote to the intermediate site.
If at the intermediate site additional volumes can be provided for a new FlashCopy relation to
the intermediate volumes, it is possible to set up an additional Global Mirror. Assuming that
the production may remain for a longer period of time at the remote site, this approach would
provide consistent data at the intermediate site and protect the production against a possible
disaster at the remote site. See 27.8, “Set up an additional Global Mirror from remote site” on
page 403, for details about how to set it up.
Example 28-14 shows how to establish the Global Copy. The -cascade option has to be
omitted because the Global Copy is now not a cascaded relation anymore. Since the volumes
at the remote and intermediate sites are identical, use a -mode nocp option in order to avoid a
FULL COPY.
Example 28-14 Establish Global Copy to the intermediate site

dscli> mkpprc -dev IBM.2107-75BYGT1 -remotedev IBM.2107-75ABTV2 -type gcp -mode nocp
7400-7403:7200-7203
IBM.2107-75BYGT1
created.
created.
created.
created.
28.4.7 Step 7: Start I/O at the remote site

At this point all necessary actions to recover the production application from the local to the
remote site have been successfully completed. The application can now be started against
the volumes at the remote site (see Figure 28-3 on page 417).
28.5 Return from remote site
The scenario described in this section includes the return from the remote site after a planned
recovery, which was discussed in 28.4, “Recovery at remote site” on page 416. This scenario
also applies after a recovery at the remote site has taken place due to a failure situation at the
local site. If the recovery at the remote site has been accomplished because of a failure
situation at the local site, the return to the local site can only be accomplished when all
required resources are available again.
As part of the recovery at the remote site scenario a Global Copy was established from the
remote to the intermediate site to ensure that the data was copied to the intermediate site
while production was running at the remote site. If this step had been omitted it must be
applied now. Ensure that all data has been drained to the intermediate site and then to the
local site before the applications are started again at the local site.
If an additional Global Mirror has been set up as described in 27.8, “Set up an additional
Global Mirror from remote site” on page 403, then it has to be removed before you proceed to
perform the steps to return to the local site.
Figure 28-4 illustrates the steps to return to the local site.
6 7
A B C
5 4 2 3
9 D
8
1
Primary Secondary
prodution production
Host host
Figure 28-4 Return from the remote to the local site
The steps to return production from remote to local are:

1. Stop I/O at remote site.
2. Fail back Metro Mirror from the intermediate site to the local site and wait until pairs are
Full Duplex.
3. Terminate Global Copy from the remote site to the intermediate site.
5. Fail over to the local site.
6. Fail back Metro Mirror from the local site to the intemediate site.
7. Establish Global Copy from the intermediate site to the remote site.

8. Start I/O at the local site.
9. Start Global Mirror or add volumes to session.
28.5.1 Step 1: Stop I/O at remote site

To return the production back to the local site, the aplications have to be stopped at a certain
point in time. It is better to stop the I/O to the remote volumes right now. This guarantees that
the data will remain identical in each location while performing this scenario.
28.5.2 Step 2: Fail back Metro Mirror from the intermediate site to the local site
The running application’s data at the remote site has been replicated because a Global Copy
was established during the failover procedure (see 28.4.6, “Step 6: Establish Global Copy
from remote to intermediate site” on page 420). When the local site becomes available again,
the Metro Mirror can be failed back.
The Metro Mirror failover described under 28.4.5, “Step 5: Failover Metro Mirror to
intermediate site” on page 419, must be issued with the option -cascade since with the fail
back to the local site, the Metro Mirror is now cascaded again to the Global Copy. This also
implies that a cascaded copy relation to the local site must not be a synchronous relation.
This is realized with the option -type gcp, which turns the Metro Mirror into Global Copy mode.
If the failover was not executed with these two options during the failover to the remote site
scenario, these options can be supplied right now with the failbackpprc command.
In Example 28-15 the failbackpprc command is issued including the options -cascade and
-type gcp. It is executed at the intermediate site. Before taking any further action, make sure
that all data was replicated to the local site. Check this with the lspprc command.
Example 28-15 Fail back Metro Mirror to local site

dscli> failbackpprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 -type gcp -cascade
7200-7203:7000-7003
IBM.2107-75ABTV2
dscli> lspprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 7200-7203:7000-7003
IBM.2107-75ABTV2
Status
===========================================================================================
=======
7200:7000 Copy Pending - Global Copy 72 300 Disabled True
28.5.3 Step 3: Terminate Global Copy from remote to intermediate site

Reversing the Global Copy using failover and failback functions would result in having the
intermediate volumes as sources for the reversed Metro Mirror and the Global Copy, which is
not permitted. For this reason the Global Copy will be terminated in this step. In a later step
(see 28.5.7, “Step 7: Create Global Copy from intermediate to remote site” on page 424) the
Global Copy is recreated from the intermediate site to the remote site in nocopy mode.
Attention: Make sure that between this step and step 6 no I/O will happen to the volumes.
Otherwise data may be corrupted due to the no copy option, and thus a full copy is
required.
Example 28-16 shows the DS CLI command to remove the Global Copy.
Important: Before the Global Copy is removed, check whether all tracks are copied to the
secondary site. Use the TSO CQUERY command to verify whether the Percent of copy
complete is 100%.
Example 28-16 Remove the Global Copy from remote to intermediate site
dscli> rmpprc -quiet -dev IBM.2107-75BYGT1 -remotedev IBM.2107-75ABTV2 7400-7403:7200-7203
IBM.2107-75BYGT1
withdrawn.
withdrawn.
withdrawn.
withdrawn.
28.5.4 Step 4: Suspend Metro Mirror

It is not necessary to suspend the Metro Mirror before the failover, because when production
will be shifted to the local site, the failback will change the status of the secondary volumes to
Host Source Suspended. The status of the primary will remain unchanged. However in
preparation of the failback, it is a good practice to suspend the Metro Mirror to bring the status
of both the primary and the secondary volumes to suspended prior to the failover.
28.5.5 Step 5: Fail over to local site

Before the failover at the local site, ensure that all tracks were transmitted to the local site.
The lspprc command issued at the intermediate site should show that there are no more
out-of-sync tracks.
Example 28-17 shows how to fail over to the local site. The command is executed at the
intermediate site.
Note: To fail over to the intermediate site you must specify the secondary volumes as the
source and the primary volumes as targets in the failoverpprc command.
Example 28-17 Failover at the local site

7000-7003:7200-7203
IBM.2107-75ABTV1

28.5.6 Step 6: Fail back Metro Mirror from the local site to the intermediate site
The failback enables tracks to be copied from the local to the intermediate site. The Metro
Mirror is now no longer in a cascaded relation, so the copy type is now mmir again. Because
the applications have not yet been started, the content of the volume at the local and the
intermediate site are the same, and the status of the Metro Mirror should be Full Duplex.
Example 28-18 shows the failback to the intermediate site at the local site.
Example 28-18 Fail back the Metro Mirror from the local site to the intermediate site
7000-7003:7200-7203
IBM.2107-75ABTV1
dscli> lspprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 -fullid -fmt default
7000-7003
IBM.2107-75ABTV1
===========================================================================================
===================================================
28.5.7 Step 7: Create Global Copy from intermediate to remote site

After the Metro Mirror has been prepared for production at the local site, the Global Copy has
to be created from the intermediate site to the remote site with the -cascade option. Since the
volumes at the remote and intermediate siteas are identical, the Global Copy should be
established with the -mode nocp option (NO COPY).
Important: Be sure that there is no active I/O to the volumes between step 3 and step 7.
Otherwise data may be corrupted due to the nocopy option and would then require a full
copy.
Example 28-19 shows the DS CLI command to create the Global Copy.
Example 28-19 Create Global Copy from the intermediate site to the remote site
dscli> mkpprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75BYGT1 -type gcp -mode nocp
-cascade 7200-7203:7400-7403
IBM.2107-75ABTV2
created.
created.
created.
created.
28.5.8 Step 8: Start I/O

Now that the Metro Mirror and Global Copy are fully re-established from the local to the
intermediate to the remote site, the application can be started at the local site.
28.5.9 Step 9: Start Global Mirror or add volumes to the session

Start the Global Mirror if it has been stopped in the corresponding recovery scenario
described in 28.4.2, “Step 2: Terminate Global Mirror or remove volumes from session” on
page 418. Or add the volumes to the session again if the Global Mirror is still up and running.
This is the case whenever the other applications are not the subject of the failover/failback
scenarios and need to have the Global Mirror to be in operation.
Example 28-20 shows how to add the volumes into the session and how to monitor the
session. Because the applications have not been started, the volumes should go immediately
into the active state.
Example 28-20 Add volumes to the session and check them

IBM.2107-75ABTV2
IBM.2107-75ABTV2
===========================================================================================
==================================
Duplex True Enable
Duplex True Enable
Duplex True Enable
Duplex True Enable

29
Chapter 29. Disaster recovery test scenarios

This chapter describes disaster recovery test scenarios. For each scenario, we describe in
detail all operations in a step-by-step approach.

29.1 Overview
Disaster recovery test scenarios are operations initiated by the user and based on
failover/failback advanced copy function features. Disaster recovery test scenarios are used
to practice readiness for a disaster. They have minimal impact on the existing Metro/Global
Mirror and no impact on the production at the local site. We describe two disaster recovery
test scenarios, fail over to the intermediate site and fail over to the remote site.
Also, all of the secnarios presented here have been practically tested and represent the best
practice for the situations they address. However, additional or alternate scenarios remain
possible depending on particular circumstances within your data center.
Note: We strongly recommend that you test all procedures extensively before deployment
to the production environment.
29.2 Disaster recovery test at the intermediate site

In this section we describe the sequence of steps required to perform a disaster recovery test
at the intermediate site, while the production keeps running at the local site. It is assumed that
the storage at the intermediate site is accessible to a host that will start up the application for
the disaster recovery test purpose.
Figure 29-1 illustrates the steps to perform a failover to the intermediate site, while the
production remains at the local site.

3 4
A B C
1
2 E D
Production 5
Host
Disaster
Recovery
Test
Host
Figure 29-1 Failover to intermediate site for disaster recovery test
The steps for this scenario of the failover to the intermediate site are as follows:
1. Prepare the failover by issuing a freeze and unfreeze of the Metro Mirror.
Note: During the freeze/unfreeze interval no data will be copied to the remote site.
2. Establish FlashCopy to the additional volumes at the intermediate site.
3. Set up PPRC paths from the local site to the intermediate site.
4. Resume Metro Mirror.
5. Start I/O at the disaster recovery host.
29.2.1 Step 1: Prepare the failover for disaster recovery test

Because the applications will continue after the failover, you must ensure that the data is
consistent at the intermediate site. To provide consistency at the intermediate site, prior to the
failover, a freeze of the Metro Mirror pairs will be issued in combination with an unfreeze (see
27.4, “Freeze and unfreeze Metro Mirror volumes” on page 398). The freeze of the Metro
Mirror will set the queue full state to the primary volumes and terminate the links between the
local and the remote site. The unfreeze that follows up immediately will remove the queue full
state from the volume, so that volumes become writable again.
Note: During the queue full state the I/O from the application is blocked and waits until the
volumes are writable again. This implies an interruption of the application I/O for the time it
takes to terminate the PPRC links. The application does not have to be stopped.
Example 29-1 shows the usage of the freezepprc and the unfreezepprc for the Metro Mirror.
Theses commands are issued at the local site. A subsequent lspprc command at the local
storage subsystem shows that the primary volumes went to a Suspended state as a result of
Freeze Metro Mirror. An lspprc command issued at the intermediate site shows the status of
the secondary volumes, which are still in Target Full Duplex mode.
Example 29-1 Freeze and unfreeze prior to the failover

At the local site:
dscli> freezepprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 70:72

IBM.2107-75ABTV1
dscli> lspprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 -fmt default 7000-7003
IBM.2107-75ABTV1
Status
===========================================================================================
=====
7000:7200 Suspended Freeze Metro Mirror 70 300 Disabled Invalid
dscli> lspprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 -fmt default 7200-7203

IBM.2107-75ABTV2
First Pass Status
===========================================================================================
==============
Invalid
Chapter 29. Disaster recovery test scenarios 429

Invalid
Invalid
Invalid
7200:7400 Copy Pending - Global Copy 72 unknown Disabled
True
True
True
True
To quickly reenable I/O to the primary volumes, the unfreezepprc command is issued
immediately after the freezepprc. Example 29-2 illustrates the unfreezepprc command.
Example 29-2 Unfreeze the primary volumes and recreate PPRC paths
dscli> unfreezepprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 70:72
IBM.2107-75ABTV1
29.2.2 Step 2: Set up FlashCopy to the additional volumes

The freezepprc has terminated the PPRC links and suspended the primary volumes. The
volumes at the intermediate site are now ready for a FlashCopy.
Example 29-3 shows the mkflash command, which is executed at the intermediate site.
Example 29-3 Set up FlashCopy on intermediate site

dscli> mkflash -dev IBM.2107-75ABTV2 7200-7203:7204-7207
IBM.2107-75ABTV2
dscli> lsflash -dev IBM.2107-75ABTV2 7200-7203:7204-7207
IBM.2107-75ABTV2
SourceWriteEnabled TargetWriteEnabled BackgroundCopy
===========================================================================================
=========================================
7200:7204 72 0 300 Enabled Disabled Disabled Disabled Enabled
Enabled Enabled
7201:7205 72 0 300 Disabled Disabled Disabled Disabled Enabled
Enabled Enabled
Enabled Enabled
Enabled Enabled
29.2.3 Step 3: Set up PPRC paths from the local site to the intermediate site
After the FlashCopy has been initiated, we can set up the PPRC paths again. More details
about the PPRC paths set up can be found in 26.3.2, “Step 1: Set up all Metro Mirror and
Global Mirror paths” on page 386. When the paths are recreated the Metro Mirror will not
copy data because the primary volumes are still in Suspend mode.
Example 29-4 shows the mkpprcpath and lspprcpath commands, which are executed on the
local site.
Example 29-4 Set up PPRC paths from the local site to the intermediate site and check the paths

5005076303FFCE63 -srclss 70 -tgtlss 72 -consistgrp I0033:I0233 I0102:I0301
IBM.2107-75ABTV1
dscli> lspprcpath -dev IBM.2107-75ABTV1 70
IBM.2107-75ABTV1
=========================================================
70 72 Success FF72 I0033 I0233 5005076303FFCE63
70 72 Success FF72 I0102 I0301 5005076303FFCE63
29.2.4 Step 4: Resume Metro Mirror

Now we can resume the Metro Mirror, which was suspended because of the freezepprc in
29.2.1, “Step 1: Prepare the failover for disaster recovery test” on page 429. Once the Metro
Mirror has resumed, the Metro/Global Mirror is active again.
Example 29-5 shows the resumepprc and lspprc commands.
Example 29-5 Resume PPRC pairs and check the state
At the local site:
dscli> resumepprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 -type mmir

7000-7003:7200-7203
IBM.2107-75ABTV1
dscli> lspprc -dev IBM.2107-75ABTV1 -remotedev IBM.2107-75ABTV2 7000-7003
IBM.2107-75ABTV1
Status
===========================================================================================
=======
7000:7200 Full Duplex - Metro Mirror 70 300 Disabled Invalid

dscli> lspprc -dev IBM.2107-75ABTV2 -remotedev IBM.2107-75ABTV1 7200-7203

IBM.2107-75ABTV2
First Pass Status
===========================================================================================
==============
Invalid
Invalid
Invalid
Invalid
True
True
True
True
29.2.5 Step 5: Start I/O on the disaster recovery host

After you have varied on the volume groups and mounted the file systems (in the case of an
AIX operating system), the applications can be started on the disaster recovery host.
When you have finished the disaster recovery test, you can unmount the file systems and
vary off the volume groups on the host and remove your FlashCopy target volumes at the
intermediate DS8000, so you are in the same state you were at the beginning of the disaster
recovery test.
29.3 Disaster recovery test at remote site

In this sectionr we describe the steps required to perform a disaster recovery test at the
remote site, while the production continues at the local site. The storage at the remote site
must be accessible to a host on which to start the application.
Figure 29-2 illustrates the steps to perform a failover to the remote site, while the production
remains at the local site.
3 4
A B C
D
2
1
E
5
Disaster
Recovery
Prodution Test
Host Host
Figure 29-2 Failover for disaster recovery test to remote
The steps are:

1. Freeze and unfreeze the Metro Mirror.
Note: During the freeze/unfreeze interval no data will be copied to the remote site.
2. Establish FlashCopy to the additional volumes at the remote site.

3. Set up PPRC paths from the local site to the intermediate site.
4. Resume Metro Mirror.
5. Start I/O at the disaster recovery host.
For this scenario it is assumed that an additional host is available to start the application from
the additional volumes that have been copied with FlashCopy at the remote site.
The steps are explained in detail in 29.2, “Disaster recovery test at the intermediate site” on
page 428. The only difference is that steps 2 and 5 will be executed on the remote site.

30
Chapter 30. Unplanned scenarios

This chapter presents a certain number of situations that can occur and impact the operations
at the different data center locations and discusses how to manage them through
Metro/Global Mirror by failover at the intermediate or the remote site. Each scenario points to
an entry into the planned scenarios described in Chapter 28, “Planned recovery scenarios”
on page 407, which must then be followed to complete the failover process.

30.1 Overview
Unplanned scenarios are operations that apply when an exceptional event affects the normal
operations at the local site and require a failover to storage resources at the intermediate or
remote site.
Depending on the kind of incident, different actions are required. It is therefore essential to
analyze the current situation very carefully in order to make the right decisions for the storage
takeover. When the situation is understood, actions should be managed in the way that one
of the planned scenarios described in Chapter 28, “Planned recovery scenarios” on
page 407, can be used to bring the system into a defined failover status.
Some situations are easier to manage when a disaster occurs. For example, when the local
site is completely destroyed, it is obvious that the storage and the applications have to be
failed over to the intermediate or the remote site. Other situations may be more complex, for
example, when some components of the local site are still in operation and other components
are damaged or inaccessible. In this case a careful analysis of the situation has to be done
before the failover processes can be accomplished.
In the following sections several types of incidents and disaster situations are discussed,
though many variations are possible.
30.2 Server outages

This situation may occur when one or more servers are down at the local site but the storage
system is not affected by the outage.
A server outage is, in general, addressed through a high availability solution like HACMP™
for AIX or other cluster solutions implemented at the local site. When the intermediate site is
not too far away from the local site and the bandwidth is high enough, a cluster across the
local and the intermediate site can be implemented as described in 26.2, “Configuration
examples” on page 383.
When there is no cluster implementation and a server outage happens, the situation can be
categorized as a disaster situation where the application and the storage have to be taken
over. This is the situation discussed in this section and illustrated in Figure 30-1.
A B C
Failover
Primary Secondary
production production
Host host
Figure 30-1 Failover after server outage at local site
In this case the storage focal point is failed over to the remote site, since the second server
will typically reside at the remote site. To achieve the failover to the remote site the scenario
described in 28.4, “Recovery at remote site” on page 416, can be used.
30.3 Link failures

A link failure is detected when a storage system cannot see its counterpart via the PPRC
links. Typically there are more than one PPRC link in use, which offers, beside the higher
bandwidth, a resiliency against single link failures. If not all links have failed, the Metro Mirror
or the Global Copy continues. As long as there are no massive performance impacts, a
failover is not needed.
30.3.1 Metro Mirror link failure

If the performance degrades or all links are broken, depending on where the secondary
server resides, a failover to the intermediate or the remote site is required. In this case the
planned scenarios in 28.2, “Recovery at intermediate site” on page 408, or in 28.4, “Recovery
at remote site” on page 416, can be used.
Using the Incremental Resync functionality it is possible to set up a Global Mirror to the
remote site, which bypasses the intermediate site, without any interruption of the data
replication. See Chapter 31, “MGM Incremental Resync” on page 443, for more information.
Chapter 30. Unplanned scenarios 437

A B C
Primary Secondary
production production
Host host
Figure 30-2 Link failure of the Metro Mirror
When all links have failed in a Metro Mirror, the pairs will go into the suspend mode after a
time-out of 30 seconds (which is the default).
To fail over to the intermediate site, see 28.2.3, “Step 3: Failover the intermediate site” on
page 410, and proceed with this step.
When a failover to the remote site has to be done, because the secondary production server
is located at the remote site, the processing should start with 28.4.2, “Step 2: Terminate
Global Mirror or remove volumes from session” on page 418, and the following steps.
If the Metro Mirror links are back before a failover has been started, the Metro Mirror pairs
have to be resumed manually with the resumepprc command.
30.3.2 Global Copy link failures

When the Global Copy links have failed completely, the copy pairs will go into the suspend
mode after the time-out of 30 seconds. The Metro Mirror is still active. A failover to the
intermediate or the remote site is not indicated. When the links are back again, the Global
Copy resumes automatically. The Global Mirror will also automatically start forming
Consistency Groups. In this case no action other than fixing the link problems is required.
30.4 Partial disasters

A disaster is always a serious impact to the infrastructure of a data center. Systems installed
in a data center can sometimes rely on redundancies provided by the data center
infrastructure. For example, server, storage, and network components are equipped, in
general, with two independent power supplies, connected to two independent power paths.
Redundancy provides an overall resilient environment for each data center, and is enough to
resist most of the incidents against single components like the servers, storage, network, or
parts of the infrastructure. But serious incidents, and usually beyond your control, such as a
massive lightning stroke, may cause an outage of more than one component of the
infrastructure. Other parts of the data center may still be faultless.
Figure 30-3 illustrates an example of a partial disaster in a data center caused by multiple
outages of the electrical power distribution.
Secondary
power
distribution
360V
4kV
SD1
Primary
power
distribution
Server 1 Storage 1
SD2
20kV
PD1
Storage 2
SD3 Server 2
Storage 3
20kV SD4
PD1
SD5
SD6
Figure 30-3 Example for a partial disaster
In this example, three secondary power distributors are out of order. The result is that all
components that are connected to SD1 or SD4 only or connected to both SD1 and SD4 have
no electrical power. All other components continue to operate.
Assume that Server1 is configured into several Logical Partitions (LPARs) and that in each
LPAR a different application or a part of an application, for example, a database engine, is
running. The storage for server 1 may be supplied by the storage subsystem Storage1,
Storage2, and Storage3.
In order to make the correct decisions for recovery, a detailed and careful analysis of this
situation is essential. Remember that not only the servers and the storage subsystems are
the subject of the recovery, but that the application must be made available at the recovery
site.
With Metro/Global Mirror you can bring applications back very quickly, and in a flexible way,
while keeping data consistent.

In our example, the following types of situations are possible:
򐂰 All servers with a total loss of power or without network access are subject to a failover.
– For the servers using Storage1, a storage failover has to execut to the intermediate or
remote site, depending on where the secondary server is installed.
– For the servers that are clustered with a server at the intermediate site and that are
supplied by another storage subsystem, a cluster failover can be applied.
– For those servers that have their secondary at the remote site, a storage failover to the
remote site has to be done.
򐂰 All servers that are still available but use Storage1 are subject to a storage failover.
– For the servers that are clustered with the intermediate site, the storage may be failed
over to the intermediate site. The server may reside at the local site, but could also be
taken over to the intermediate site as well.
– For the servers having their secondary at the remote site, a storage failover to the
remote site has to be done.
Based on the conclusions made above, the following decision table may help you to find the
correct scenario for the Metro/Global Mirror operations.
Table 30-1 Failover decision table

Server Storage Location of Apply Apply Applicable
status status secondary storage server scenario
server failover failover starting at
Running Available Intermediate N/A N/A N/A
Running Available Remote N/A N/A N/A
Running Failed Intermediate To interm. Yes/noa 28.2.3, “Step

3: Failover the
intermediate
site” on
page 410
Running Failed Remote Remote Implicitb 28.4.2, “Step

2: Terminate
Global Mirror
or remove
volumes from
session” on
page 418
Failed Available Intermediate Yes/noc Yes 28.2.3, “Step

3: Failover the
intermediate
site” on
page 410
Failed Available Remote To remote Implicit 28.4.2, “Step

2: Terminate
Global Mirror
or remove
volumes from
session” on
page 418
Server Storage Location of Apply Apply Applicable
status status secondary storage server scenario
server failover failover starting at
Failed Failed Intermediate To interm. Yes 28.2.3, “Step

3: Failover the
intermediate
site” on
page 410
Failed Failed Remote To remote Implicit 28.4.2, “Step

2: Terminate
Global Mirror
or remove
volumes from
session” on
page 418
a. Depends on the decision of the client, when the server is clustered to the intermediate site.
b. A storage failover to the remote site implies a server failover to the remote site, since there
is no stretched cluster between the local and remote site.
c. Not recommended when the server is clustered to the intermediate site.
A partial disaster, as can occur in large data centers, is the most challenging kind of disaster
because of the many different situations that can result. The complexity is apparent even
when only 20% of the applications are affected. This percentage is too low to fail over a
complete data center. But when a data center hosts 100 applications, the client has to
manage 20 different critical situations.
30.5 Data center outages

A data center outage is when a massive external incident happens that completely impacts
the data center. This does not necessarily mean that the center is completely destroyed, but
that all the essential infrastructure components, like electrical power, network, storage, or the
data center air conditioning have completely failed.
For a Metro/Global Mirror implementation, disaster recovery is only indicated when there is an
outage of the data center at the local site. In this case an unplanned failover scenario like that
described in 30.3.1, “Metro Mirror link failure” on page 437, can be applied.
When the intermediate or the remote site is down, the production at the local site can still
continue. Only the disaster recovery capabilities are degraded. When the remote site is down,
Metro Mirror to the intermediate site is still available.
When the intermediate site is down and there is a connection with sufficient bandwidth
available between the local and the remote site, a Global Mirror to the remote site can be set
up. In this case a Global Copy between the Metro Mirror primary volumes and the Global
Copy secondary volumes can be set up. This requires that at the local and the remote site the
current pair relation is removed using the rmpprc command with the options -at src and
-unconditional. The links between the storage subsystems at the local and the remote site
have to be established and the paths have to be created according to 26.3.2, “Step 1: Set up
all Metro Mirror and Global Mirror paths” on page 386. The steps required to set up the Global
Mirror are described in 26.3.3, “Step 2: Set up Global Copy NOCOPY from intermediate to
remote sites” on page 388, and in 26.3.6, “Step 5: Create Global Mirror session and add
volumes to session” on page 389, and the following sections.

When there is a mix of primary and secondary production between the local and the
intermediate site, as shown in the example configurations in Chapter 26, “Configuration and
setup” on page 381, the applications of the failed data center have to be failed over to their
secondary site. The Global Mirror for these applications is already established because it is
part of the normal setup of the Metro/Global Mirror.
31
Chapter 31. MGM Incremental Resync

This chapter explains and illustrates the Incremental Resynchronization (or Incremental
Resync) feature now available in the context of a Metro/Global Mirror environment.
The chapter starts with an overview and functional description of Incremental Resync and
explains the new DSCLI options that support it. This is followed by a section explaining how
to set up Incremental Resync for Metro/Global Mirror.
Finally, this chapter includes scenarios with detailed operations presented in a step-by-step
approach and covering the following situations:
򐂰 Failure at the local site
򐂰 Failure at the intermediate site
򐂰 Returning to normal operations

31.1 Overview
We know that in a Metro/Global Mirror environment data is copied from the local to the
intermediate site and then cascaded from the intermediate site to the remote site. Obviously,
if there is a storage failure (or disaster) at the intermediate or local site, or even simply a loss
of connectivity between the local and intermediate sites, data can no longer be cascaded to
the remote site. However, if we establish physical connectivity as well directly between the
local and remote sites, we could in case of failure at the intermediate site still copy data from
the local to the remote site. Incremental Resync gives the capability to use this connection
with a Global Mirror relationship between the local and remote to copy data after a failure at
the remote site, without having to recopy all the data. Figure 31-1 shows how the Incremental
Resync is used.
Local site Intermediate site Remote site
Global Mirror
DS8000 DS8000 DS8000
Metro Mirror Global Mirror
2109-M14
2109-M14
2109-M14
P590
P590
P550 P550
Figure 31-1 Setup of a Metro/Global Mirror for Incremental Resync
31.1.1 Functional description

A prerequisite to Incremental Resync is to have established paths from the local site to the
remote site. These paths are required for the Global Mirror which will be created when the
failing intermediate site is bypassed. Depending on the nature of the failure at the
intermediate site, it could be that the Global Mirror between the local and remote site will
need to be used for an extended period of time. To be prepared for future failover and failback
operations we recommend that you establish the paths in both directions.
To enable the Incremental Resync capability a specific option must be specified when initially
establishing the Metro Mirror relationship between the local and intermediate sites. If the
Metro Mirror is already established and in full duplex, the Incremental Resync is enabled by
issuing the mkpprc command against the existing relation with the specific option to enable
Incremental Resync.
When you specify Incremental Resync, in addition to the existing Change Recording bitmap
of the Global Mirror at the intermediate site, another Change Recording bitmap is created at
the Metro Mirror primary volumes. This bitmap keeps track of any incoming I/O while a
consistency group is copied from the intermediate site until it has arrived to the FlashCopy
target volumes at the remote site. The Global Mirror at the intermediate site is queried
periodically for the current status of the tracks between the intermediate and the remote site.
The change recording will restart from a cleared change recording bitmap, when the next
consistency group is formed at the intermediate site. Figure 31-2 shows an overview of the
different phases of the Incremental Resync.
Done Done Done Done time

Start Start Start Start
Metro Mirror GM Coord, time GM Drain time GM FlashCopy Phases
C CO
R RO
Incremental Resync S
bitmap Global Mirror
bitmaps
querying
A B C
Figure 31-2 Incremental Resync phases
To illustrate the functionality of Incremental Resync, a failure of the intermediate side is

assumed, which means that no more data can be copied to the remote site.
If there is a failure of the intermediate site, the Metro Mirror primary volumes will go into
suspend mode. With Incremental Resync enabled, the tracks that could not be copied to the
intermediate site are recorded in the change recording bitmap at the primary site.
To bypass the failed intermediate site, a new Global Copy relationship should now be
established from the local volumes to the FlashCopy source volumes at the remote site. At
this point in time, however, these volumes are still Global Copy target volumes of the relation
from the intermediate site. Therfore, failover of the existing Global Copy at the remote site
must take place before the new Global Copy between the local and remote sites can be
established.
Chapter 31. MGM Incremental Resync 445

The new Global Copy from the local site to the remote site must be established with the
Incremental Resync option. As a result, all the tracks recorded in the change recording
bitmap and in the out-of-sync recording of the Metro Mirror are copied to the remote site.
When all out-of-sync tracks have been sent from the local to the remote site, a Global Mirror
can be started at the local site using the remote volumes as the Global Mirror target volumes
(see Figure 31-3).
A B C
Primary
production
host
Figure 31-3 Incremental Resync overview
When the intermediate site is available again the Metro/Global Mirror must be re-configured
back to normal operation.
First of all, the intermediate site has to be checked for its current status of the relations. The
former Global Mirror from the intermediate site to the remote site must be removed. To copy
data missing at the intermediate site since the failure, a failback from the remote site to the
intermediate site is accomplished.
When the incremental copy is completed, the procedure to return to normal operations from
local to remote via the intermediate site can be initiated.
First, the Global Mirror and the Global Copy relationships to the remote site have to be
removed. Next, the Global Copy between the intermediate and the remote site has to be
re-established in its original direction. Then the Metro Mirror can be recreated using the
Incremental Resync option. The Metro Mirror will copy only the data changed since the
Global Copy from the local to the remote site was removed. The final step is to recreate the
Global Mirror at the intermediate site, which enables consistency groups to be formed at the
intermediate and drained to the remote site.
31.1.2 Options for DSCLI

The Incremental Resync is enabled using the mkpprc command with the option
-incrementalresync. The option requires the following values to specify the different
parameters of Incremental Resync:
enable This enables the Incremental Resync functionality. It creates initialized
change recording bitmaps. (Bitmaps consist initially of all ones. As
consistency groups are formed in Global Mirror the bitmaps will
change to reflect only the changed data.)
enablenoinit This enables the Incremental Resync functionality and creates change
recording bitmaps. The bitmaps will not be initialized. That is, they will
all be zeroes. This option should only be used in certain cases as part
of the return scenario for Incremental Resync.
disable This stops the Incremental Resync mechanism.
recover This brings in relations of local to remote volumes as new pairs. The
devices specified in the command are checked to ensure that they
currently have a target device in common. In other words, it verifies
that the devices are part of an existing Metro/Global Mirror
relationship.
override This is the same as above, but the volumes may not be part of a
Metro/Global Mirror. As opposed to a recover, in this case the
checking is not done
31.2 Setting up Metro/Global Mirror with Incremental Resync

This section describes how to set up Metro/Global Mirror with incremental re-synchronization,
either from scratch or from an existing two-site Global Mirror.
򐂰 The initial setup procedure (from scratch) assumes that no existing Metro Mirror or Global
Copy relationships.
򐂰 The setup procedure from an existing Global Mirror environment explains how to set up
Metro/Global Mirror with Incremental Resync environment by adding an intermediate site.
31.2.1 Setup of Incremental Resync Metro/Global Mirror

To set up Incremental Resync Metro/Global Mirror, follow the steps in 26.3, “Initial setup of
Metro/Global Mirror” on page 385.
The steps must be executed in the same order. However, 26.3.4, “Step 3: Set up Metro Mirror
between local and intermediate sites” on page 388, is modified as follows to implement
Metro/Global Mirror with Incremental Resync.
Specify the -incrementalresync enable option in conjunction with -type mmir options when
executing the mkpprc command at the local site to create the Metro Mirror relationship. This
enables the Incremental Resync function and creates change recording bitmaps for the Metro
Mirror primary volumes running at the local site.
When the Metro Mirror is already established, but Incremental Resync is not yet enabled, the
option -mode nocp is required to enable Incremental Resync.
Example 31-1 shows how to set up incremental re-synchronization for Metro/Global Mirror.
Example 31-1 Setup of Incremental Resync Metro/Global Mirror

dscli> mkpprc -dev IBM.2107-1301651 -remotedev IBM.2107-1301261 -type mmir -mode nocp -incrementalresync enable 2000-2007:2000-2007
dscli>

dsclI> lspprc -dev IBM.2107-1301651 -l 2000-2007
Critical Mode First Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG
==========================================================================================================================================
=========================================================================
2000:2000 Full Duplex - Metro Mirror 0 Disabled Disabled Invalid - 20 300
Disabled Invalid Enabled Disabled Disabled Enabled
dscli>
dscli>
The remaining steps in 26.3, “Initial setup of Metro/Global Mirror” on page 385, are followed
to complete the setup for Metro/Global Mirror with incremental resync.
31.2.2 Going from Global Mirror to Incremental Resync Metro/Global Mirror

Incremental Resync Metro/Global Mirror can be initialized if production is already running in a
Global Mirror environment. The transition from the Global Mirror environment to an
Incremental Resync Metro/Global Mirror environment is performed with a full copy from the
remote site to the intermediate site. In this scenario the new site will be the intermediate site.
Figure 31-4 illustrates the steps to transition from a current Global Mirror environment to a
Metro/Global Mirror with Incremental Resync environment.
Primary
production Local Site
host
A
3 Remote Site
4
1
7
2
Intermediate Site C
D
6 5
B 1
Figure 31-4 Moving from Global Mirror to Incremental Resync Metro/Global Mirror
The steps for setting up and initializing Metro/Global Mirror with Incremental Resync if Global
Mirror is already running are as follows. (The assumption is that there is an existing Global
Mirror between the A and C volumes, and an intermediate site (B volumes) is added to the
configuration.)
1. Set up all PPRC paths.
2. Start Global Copy from the remote to the new intermediate site.
3. Start incremental re-synchronization from the local site.
4. Terminate Global Mirror and suspend Global Copy at the local site.
5. Terminate locate to remote Global Copy local to remote at remote site.
6. Reverse Global Copy to run from intermediate to remote site.
7. Set up Metro Mirror from local to intermediate site.
8. Set up Global Mirror at intermediate site.
Step 1: Set up all PPRC paths

Before setting up the Metro/Global Mirror with a new intermediate site, all paths should be
established. Before establishing PPRC paths, the PPRC ports should first be identified. See
26.3.1, “Identifying the PPRC ports” on page 386, to determine which PPRC ports are
available and can be used as links for the Metro/Global Mirror environment.
Once the PPRC ports are identified, PPRC paths can then be created from the local to the
intermediate site and another from the intermediate to the remote site. It is also good practice
to create the PPRC paths in the opposite directions, from the intermediate to the local site
and from the remote to the intermediate site. There should be a total of four new PPRC paths
created. Prerequisites for creating a successful PPRC path and reasons for creating PPRC in
both directions are explained in 26.3.2, “Step 1: Set up all Metro Mirror and Global Mirror
paths” on page 386.
To create the new PPRC paths, the command mkpprcpath is executed from each of the sites.
Step 2: Start Global Copy from the remote to the intermediate site
To migrate the data from the remote site to the intermediate site, Global Copy is started. This
will take the data that is currently running from the local to remote and copy it (or rather
cascade it) to the intermediate site.
The mkpprc command with the option -type gcp is executed from the remote site to the
intermediate site. Because there is already a Global Copy relationship from the local to the
remote site, the -cascade option must also be used with the mkpprc command. This command
will create a Global Copy relationship from the remote site volumes that are currently Global
Copy secondaries to the intermediate site volumes.
Example 31-2 shows the DSCLI command to create the Global Copy. The subsequent lspprc
command shows the cascading to from local to remote to intermediate.
Example 31-2 Start Global Copy from the remote to the intermediate site
dscli> mkpprc -dev IBM.2107-75DNXC1 -remotedev IBM.2107-7520781 -type gcp -mode full -cascade 6000-6003:6600-6603
dscli>
dscli> lspprc -dev IBM.2107-75DNXC1 -fullid 6000-6003
ID State Reason Type SourceLSS Timeout (secs) Critical Mode First
Pass
Status====================================================================================================================================
=================
IBM.2107-7503461/6200:IBM.2107-75DNXC1/6000 Target Copy Pending - Global Copy IBM.2107-7503461/62 unknown Disabled
Invalid
Invalid

Invalid
Invalid
IBM.2107-75DNXC1/6000:IBM.2107-7520781/6600 Copy Pending - Global Copy IBM.2107-75DNXC1/60 unknown Disabled False
dscli>
dscli>
Step 3: Start incremental re-synchronization from the local site

Before terminating the original Global Mirror that is running from the local to the remote site,
Incremental Resync should be enabled at the local site. This takes the current Global Mirror
relationship and creates change recording bitmaps at the local site to keep track of all
updates occurring from production at the local site. The updates at the local site will be
needed when Metro Mirror with Incremental Resync is established later in “Step 7: Set up
Metro Mirror from the local to the intermediate site” on page 452.
The mkpprc command with the option -incrementalresync enablenoinit is executed from the
local site using the current Global Mirror relationship from the local to the remote site as
Example 31-3 Start incremental re-synchronization from the local site

dscli> mkpprc -dev IBM.2107-7503461 -remotedev IBM.2107-75DNXC1 -type gcp -mode nocp -incrementalresync enablenoinit 6200-6203:6000-6003
dscli>
dscli> lspprc -dev IBM.2107-7503461 -l 6200-6203
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs) Critical Mode Fi
rst Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG
======================================================================================================================================================
==============================================================
6200:6000 Copy Pending - Global Copy 0 Disabled Enabled Invalid - 62 120 Disabled Tr
ue Enabled Disabled Disabled Disabled
dscli>
dscli>
Step 4: Terminate Global Mirror and suspend Global Copy at local

Now that the current active volumes are being updated according to the change recording
bitmaps on the local site, Global Mirror can be terminated at the local site. This will cause the
FlashCopy targets on the remote site to age. The rmgmir command is executed at the local
site to terminate the Global Mirror from the local to remote site.
The pausepprc command is used to suspend the Global Copy relationship from the local site
to the remote site. By suspending the Global Copy relationship, the local and remote site
volumes will both be in a suspended state and the change recording bitmaps will remain
active at the local site.
By suspending Global Copy from the local to the remote site, data at the intermediate site will
no longer be in sync with data at the local site. Once Global Copy is suspended, data is no
longer moving to the intermediate site via the remote site. Therefore the data at the
intermediate site begins to age as production continues running at the local site. The change
recording bitmaps at the local site are keeping track of all updates from production.
In Example 31-4 the Global Mirror and the sessions are removed at the local site. Finally the
Global Copy from local to remote is suspended.
Example 31-4 Terminate Global Mirror and suspend Global Copy at local
dscli> rmgmir -dev IBM.2107-7503461 -quiet -lss 62 -session 2
dscli> chsession -dev IBM.2107-7503461 -lss 62 -action remove -volume 6200-6203 2
dscli> rmsession -dev IBM.2107-7503461 -quiet -lss 62 2
dscli>
dscli> pausepprc -dev IBM.2107-7503461 -remotedev IBM.2107-75DNXC1 6200-6203:6000-6003
dscli>
dscli>
Step 5: Terminate Global Copy at the remote site

Even though the Global Copy has been suspended from the local to the remote site, the
volumes at the remote site are still Global Copy secondaries. For the remote to lose
knowledge of being a secondary of the local, Global Copy is terminated at the remote site.
The command rmpprc with the -unconditional -at tgt option is executed at the remote site. This
termination does not affect the local site’s state. Therefore it allows for the out-of-sync
bitmaps to remain in operation at the local site. The state of the local will remain primary
suspended while the remote will no longer show as suspended target.
This step is necessary to allow the failback of the intermediate to the remote site when
reversing Global Copy in the next step“Step 6: Reverse Global Copy to run from intermediate
to remote site” on page 452.
Note: All remaining out-of-sync tracks are transferred at this point from the remote to the
intermediate site and must be drained before proceeding to the next step. To query
out-of-sync tracks, issue the command lspprc -l at the remote site. Once the out-of-sync
tracks are drained, the remote site and the intermediate site will be in sync.
In Example 31-5 the DSCLI command to terminate the Global Copy at the remote site is
shown. The lspprc command at the local site shows that the relations at this site are
untouched, while at the remote site only the Global Copy from remote to intermediate exists.
Example 31-5 Terminate Global Copy local to remote at remote site

dscli> rmpprc -quiet -dev IBM.2107-75DNXC1 -unconditional -at tgt 6000-6003
CMUC00155I rmpprc: Remote Mirror and Copy volume pair :6000 relationship successfully withdrawn.
dscli>
dscli> lspprc -dev IBM.2107-7503461 -remotedev IBM.2107-75DNXC1 -fullid -fmt default 6200-6203
Status
==========================================================================================================================================
======
IBM.2107-7503461/6200:IBM.2107-75DNXC1/6000 Suspended Host Source Global Copy IBM.2107-7503461/62 120 Disabled True
dscli>

# At remote site
#
dscli> lspprc -dev IBM.2107-75DNXC1 -fullid 6000-6003

Status
==========================================================================================================================================
====
IBM.2107-75DNXC1/6000:IBM.2107-7520781/6600 Copy Pending - Global Copy IBM.2107-75DNXC1/60 120 Disabled True
dscli>
dscli>
Step 6: Reverse Global Copy to run from intermediate to remote site

Once the out-of-sync tracks are drained, the Global Copy relationship initially established
from the remote to the intermediate site can be reversed. To reverse the Global Copy
relationship a failover and failback must first take place.
To fail over to the intermediate, the failoverpprc command with the option -type gcp is
executed at the intermediate site. Because the Global Copy that is currently running from the
remote to the intermediate is cascaded, the -cascade option is also specified with the
failoverpprc command. Once the failoverpprc command is executed with the -type gcp
and -cascade options, the remote site volumes status will become primary suspended.
Next is to fail back Global Copy at the intermediate site by using the failbackpprc command
with the option -type gcp at the intermediate site. Once again, since Global Copy is in a
cascaded relation, the -cascade option is also specified.
Failover and failback has successfully reversed the Global Copy relationship. Now Global
Copy can be started from the intermediate site. To start Global Copy, the mkpprc command
with the -type gcp and -cascade options is executed. Once again, the -cascade option is used
in preparation for the next step, where the Global Copy primary is also going to be a Metro
Mirror secondary.
Step 7: Set up Metro Mirror from the local to the intermediate site
Metro Mirror from the local to the intermediate site can now be created. The command mkpprc
with the -type mmir option is executed at the local site. In “Step 3: Start incremental
re-synchronization from the local site” on page 450 change recording bitmaps were created
to keep track of all updates from production. To recover and restore these updates, the
-incrementalresync recover option is specified with the mkpprc command. The recover
parameter of the -incrementalresync will establish the Metro Mirror relationship after doing a
check at the intermediate site (the Metro Mirror secondary) for a relationship. The change
recording bitmaps created in “Step 3: Start incremental re-synchronization from the local site”
on page 450 are then merged with the out-of-sync bitmaps at the local site.
Note: To prepare for a disaster, the next step would be to start Metro Mirror with
Incremental Resync enabled. The command mkpprc with the options -type mmir and
-incrementalresync enable would be executed at the local site. This creates new change
recording bitmaps for all the Metro Mirror primary volumes.
Note: Metro Mirror will need to become full duplex and Global Copy will need to complete
first pass. To query the status of Global Mirror and Metro Mirror, use the lspprc -l
command at the local site to query Metro Mirror and at the intermediate site to query
Global Copy.
In Example 31-6 the Metro Mirror created in the first step with the option -incrementalresync
recover, which initiates copying all tracks that are marked in the change recording bitmap at
the local site. Once all the tracks have drained to the intermediate site the incremental
re-synchronization is restarted.
Example 31-6 Set up Metro Mirror from local to intermediate site

dscli> mkpprc -dev IBM.2107-7503461 -remotedev IBM.2107-7520781 -type mmir -mode full -incrementalresync recover 6200-6203:6600-6603
dscli>
dscli> lspprc -dev IBM.2107-7503461 -l -fullid 6200-6203
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date
Suspended SourceLSS Timeout (secs) Critical Mode First Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG
==========================================================================================================================================
======================================================================================================================
IBM.2107-7503461/6200:IBM.2107-7520781/6600 Full Duplex - Metro Mirror 0 Disabled Enabled Invalid -
IBM.2107-7503461/62 120 Disabled Invalid Disabled Disabled Disabled Disabled
dscli>
dscli>
dscli>lspprc -dev IBM.2107-7503461 -l 6200-6203
==========================================================================================================================================
==========================================================================
6200:6600 Full Duplex - Metro Mirror 0 Disabled Enabled Invalid - 62 120
Disabled Invalid Enabled Disabled Disabled Disabled
dscli>
dscli>
Step 8: Set up Global Mirror at the intermediate site

Once Metro Mirror is full duplex, then Global Mirror can be started at the intermediate site and
consistency groups will start forming successfully. Global Mirror is started by executing the
command mkgmir at the intermediate site.
Important: Verify that the out-of-sync tracks have completed for Metro Mirror before
starting Global Mirror. Otherwise the consistency groups will start to fail when Global Mirror
is started with the mkgmir command.
Example 31-7 shows the steps to setup the Global Mirror. The showgmir commands at the
end verify that the Global Mirror is forming consistency groups.
Example 31-7 Set up Global Mirror at intermediate site

dscli> mksession -dev IBM.2107-7520781 -lss 66 2
dscli>
dscli> chsession -dev IBM.2107-7520781 -lss 66 -action add -volume 6600-6603 2
dscli>
dscli> mkgmir -dev IBM.2107-7520781 -lss 66 -session 2
dscli>

dscli> lssession -dev IBM.2107-7520781 66 2
=============================================================================================================================
66 02 CG In Progress 6600 Active Primary Copy Pending Secondary Full Duplex True Enable
dscli>
dscli> showgmir -metrics 66
ID IBM.2107-7520781/66
Last Successful CG Form Time 11/17/2006 23:20:06 CET
First Failure LSS -
Last Failure LSS -
dscli>
ID IBM.2107-7520781/66
Last Successful CG Form Time 11/17/2006 23:20:19 CET
First Failure LSS -
Last Failure LSS -
dscli>
dscli>
31.3 Failure at the local site scenario

The failure at the local site scenario is used when the local site becomes unavailable or is
inaccessible to the Metro/Global Mirror session. The parts that are described for this scenario
are:
򐂰 The loss of the local site or a planned outage, where production is moved to the
intermediate site and Global Mirror continues from the intermediate to the remote site in a
two site environment.
򐂰 Re-introducing the local site when it is brought back, which will take the two-site
environment and move it back to a Metro/Global Mirror environment with production
remaining at the intermediate site.
򐂰 Moving production back to the local site and going back to the original Metro/Global Mirror
configuration.
31.3.1 Local site fails

This scenario starts with Metro/Global Mirror with Incremental Resync running from local to
intermediate and cascading from intermediate to the remote site. When there is an unplanned
or planned outage at the local site, the Metro/Global Mirror can then be transitioned to a
two-site mode running from the intermediate to the remote site. To get to this state, a series of
tasks must be performed. The corresponding steps are first outlined without a detailed
description or illustration of the commands. The steps are depicted in Figure 31-5.
Local Site
Remote Site
A
Primary
production
1
host
2
C
Intermediate Site
Secondary
production D
host
Figure 31-5 Local site failure
The following steps must be performed to transition to a two- site environment after the local
site failure:
1. Fail over to the remote site.
2. Terminate Metro Mirror at the intermediate site.
3. Start applications at the intermediate site.
Step 1: Fail over to the remote site

First, a failover with the force command is executed to the remote site devices. This allows for
changes to be tracked for later re-synchronization when going back to the local site devices.
The failover with force option means that there is no validation of the remote site devices to
check that they are secondary devices from the local site devices. Doing this will later allow us
to make the remote site the primary for the local site devices (see “Step 1: Failback Global
Copy from remote to local site” on page 458). When this is done the Global Copy from remote
to intermediate site will be a cascade relation to the Global Copy from Intermediate to remote.
Thus the -cascade option must be specified in this step as well.

Example 31-8 shows the execution of the failoverpprc command with the -force and
-cascade options. The lspprc command, issued at the remote site, shows that now the
Global Copy is cascaded to the Global Copy from intermediate to remote.
Example 31-8 Forced failoverpprc command from local to remote

dscli> failoverpprc -dev IBM.2107-1300561 -remotedev IBM.2107-1301651 -type gcp -cascade -force 2000-2007:2000-2007
dscli>
dscli> lspprc -dev IBM.2107-1300561 -fullid 2000-2007
ID State Reason Type SourceLSS Timeout (secs) Critical Mode First Pass S
tatus
=================================================================================================================================================
=====
IBM.2107-1300561/2000:IBM.2107-1301651/2000 Suspended Host Source Global Copy IBM.2107-1300561/20 unknown Disabled True
IBM.2107-1301261/2000:IBM.2107-1300561/2000 Target Copy Pending - Global Copy IBM.2107-1301261/20 unknown Disabled Invalid
dscli>
dscli>
Step 2: Terminate Metro Mirror at intermediate site

Next is to terminate Metro Mirror at the intermediate site. By terminating Metro Mirror, the
intermediate site devices will no longer have any knowledge of being a secondary device from
the local site devices, which will allow production to be started and recovered at the
intermediate site.
In Example 31-9 the DSCLI command to terminate the Metro Mirror at the intermediate site is
shown. With the following lspprc command issued at the intermediate site, you can see that
only the relation to the remote site remains.
Example 31-9 Terminate Metro Mirror at intermediate site

dscli> rmpprc -quiet -dev IBM.2107-1301261 -unconditional -at tgt 2000-2007
dscli>
Status
==========================================================================================================================================
====
IBM.2107-1301261/2000:IBM.2107-1300561/2000 Copy Pending - Global Copy IBM.2107-1301261/20 120 Disabled True
dscli>
dscli>
Step 3: Start applications to the intermediate site
Production can now be moved from the local site to the intermediate site and applications can
be started. The out-of-sync bitmaps will have continued to be in use. Global Mirror will
continue to operate from the intermediate site to the remote site until the local site is brought
back and made available again.
31.3.2 Local site is back

When the local site is available again, then Metro/Global Mirror with Incremental Resync can
be started again. Once the two-site mode is transitioned back to Incremental Resync
Metro/Global Mirror, production will remain running at the intermediate site. In addition,
Metro/Global Mirror will be running with Metro Mirror from the intermediate to the local site
and Global Copy from the intermediate to the remote site.
Figure 31-6 illustrates the steps to start up Metro/Global Mirror with Incremental Resync
when the local site is available to use.
Local Site
Remote Site
1
8 A 4
C
Intermediate Site 7
D
3
B
Secondary
production
host
Figure 31-6 Local site is back
The following steps are taken when re-introducing the local site and moving back to a
Metro/Global Mirror with Incremental Resync environment:
1. Fail back Global Copy from remote to local site.
2. Start incremental re-synchronization at the intermediate site.
3. Terminate Global Mirror and suspend Global Copy from the intermediate to the remote
site.
4. Suspend Global Copy from the remote site to the local site.
5. Terminate Global Copy at the remote site.

6. Reverse Global Copy to run from the local site to the remote site.
7. Start Metro Mirror from the intermediate site to the local site.
8. Start Global Mirror at the local site.
Step 1: Failback Global Copy from remote to local site

Once the local site is available again and all relationships are cleaned up, re-synchronization
of the local site can be started by starting a Global Copy from remote to local (this is possible
because of the forced failover previously executed from the remote to the local site; see “Step
1: Fail over to the remote site” on page 455). The Global Mirror from the intermediate to the
remote site will provide the recovery capability to resynchronize the local site. The local site’s
out-of-sync bitmaps are used to re-synchronize changes that may have occurred during the
failure. The remote site’s out-of-sync bitmaps contain changes made after the failure. There
may still be changes at the intermediate site that have not made it to the remote site yet. In
this case, a full copy is performed.
Note: Remote to local site out-of-sync tracks will need to be drained completely before
continuing to the next step. This will ensure that the local site has been fully
re-synchronized.
Tip: When the local site is back, in addition to cleaning up the remaining relations, it may
also be necessary to clear remaining SCSI reservations from the host access before the
failure occurred.
Example 31-10 shows the command execution.
Example 31-10 Failback Global Copy from remote to local site

dscli> failbackpprc -dev IBM.2107-1301651 -remotedev IBM.2107-1300561 -type gcp -cascade
2000-2007:2000-2007
dscli>
dscli>
Step 2: Start incremental re-synchronization at the intermediate site

Before terminating the Global Mirror relationship from the intermediate to the local site, the
incremental re-synchronization command is enabled without initialization from the
intermediate to the local site. This utilizes the current Global Mirror relationship and creates
change recording bitmaps at the intermediate site to keep track of all updates occurring from
production at the intermediate site. The updates at the intermediate site will be used when the
local site is re-synchronized after the remote is no longer getting data from the intermediate,
as we will see in “Step 7: Start Metro Mirror from intermediate to local site” on page 462.
Example 31-11 shows the command to enable incremental re-synchronization at the
intermediate site. To verify, the lssprc -l command is issued.
Example 31-11 Start incremental re-synchronization at intermediate site

dscli>
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs) Critical Mode F
irst Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG
=====================================================================================================================================================
===============================================================
2000:2000 Full Duplex - Metro Mirror 0 Disabled Enabled Invalid - 20 120 Disabled I
nvalid Enabled Disabled Disabled Disabled
dscli>
dscli>
Step 3: Terminate Global Mirror and suspend Global Copy

Now that the current active volumes are being updated according the change recording
bitmaps on the intermediate site, Global Mirror can be terminated at the intermediate to
remote site. When this scenario is executed with volumes that belong to one or more
application hosts, but not for all volumes, the Global Mirror may not be stopped, because it is
used to form consistency groups to the remote site for the remaining applications. In this case
the volumes are removed from the session of the Global Mirror. The consistency groups of
those volumes at the FlashCopy targets on the remote will begin to age.
In addition, to stop data transfer to the remote site, the Global Copy relationship from the
intermediate to the remote site is suspended. By suspending the Global Copy relationship,
the intermediate and remote sites’ volumes will both be in a suspended state and the change
recording bitmaps will remain active at the intermediate site.
In Example 31-12 the volumes of a certain application host are removed from the session. To
ensure that a final consistency group is formed, the Global Mirror is paused first. When the
volumes have been removed from the session the Global Mirror is resumed. Finally, the
Global Copy intermediate to remote is suspended.
Example 31-12 Removing volumes from the session and suspending the Global Copy intermediate to remote
dscli> pausegmir -dev IBM.2107-1301261 -lss 20 -session ad
CMUC00163I pausegmir: Global Mirror for session ad successfully paused.
dscli>
dscli> chsession -dev IBM.2107-1301261 -lss 20 -action remove -volume 2000-2007 ad
CMUC00147I chsession: Session ad successfully modified.
dscli>
dscli> resumegmir -dev IBM.2107-1301261 -lss 20 -session ad
CMUC00164I resumegmir: Global Mirror for session ad successfully resumed.
dscli>
dscli> lssession -dev IBM.2107-1301261 20 ad
========================================================================================================
20 AD - - - - - - -

dscli>
dsclI> pausepprc -dev IBM.2107-1301261 -remotedev IBM.2107-1300561 2000-2007:2000-2007
dscli>
dscli> lspprc -dev IBM.2107-1301261 -remotedev IBM.2107-1300561 -fullid -fmt default 2000-2007
Status
==========================================================================================================================================
======
IBM.2107-1301261/2000:IBM.2107-1300561/2000 Suspended Host Source Global Copy IBM.2107-1301261/20 120 Disabled True
dsclI>
dsclI>
Step 4: Suspend Global Copy remote to local site

In preparation for reversing Global Copy at the remote site, the Global Copy from the remote
to the local site that was created in “Step 1: Failback Global Copy from remote to local site” on
page 458 is suspended. This should be done once the remote to local site out-of-sync
bitmaps are completely drained from the last of the updates at the remote site.
Note: Remote to local site out-of-sync tracks must be drained completely before continuing
to the next step. This will ensure that the local site has been fully re-synchronized.
In Example 31-13 the command to suspend the Global Copy from remote to local is shown.
Example 31-13 Suspend Global Copy from remote to local site

dscli> pausepprc -dev IBM.2107-1300561 -remotedev IBM.2107-1301651 2000-2007:2000-2007
dscli>
dscli>
Step 5: Terminate Global Copy intermediate to remote at the remote site

Although Global Copy was suspended from the intermediate to the remote site, the remote
site volumes still have the status of Global Copy secondary devices. For the remote site
devices to lose knowledge of being a secondary of the intermediate, the intermediate to
remote Global Copy relationship is terminated at the remote site. This termination does not
change the intermediate site’s state and therefore allows for the out-of-sync bitmaps to
remain in operation at the intermediate site. The state of the intermediate will remain primary
suspended while the remote will no longer show as suspended target and will be terminated.
This step is necessary to allow the failback of the intermediate to the remote site when
reversing Global Copy in “Step 6: Reverse Global Copy to run local to remote site” on
page 461.
Example 31-14 shows how to terminate the Global copy at the remote site.
Example 31-14 Terminate Global Copy from intermediate to remote at remote

dscli>
dscli>
Step 6: Reverse Global Copy to run local to remote site

Once the Global Copy out-of-sync tracks are transferred and drained to the local site, then the
Global Copy relationship from the remote to the local site can be reversed. To reverse the
Global Copy relationship a failover and failback must be done before creating the Global Copy
in the reverse direction.
The local to remote failover is executed at the local site and using cascading Global Copy
mode options. The failover will change the local site volumes status to become primary
suspended. Next the failback Global Copy local to remote site is executed at the local site with
cascading allowed and mode Global Copy.
A successful failover and failback reverses the Global Copy, which now runs from the local to
the remote site. Global Copy can be established to run from the local to the remote site with
Global Copy mode and cascading allowed.
Example 31-15 shows how to reverse the direction of the Global Copy between local and
remote.
Example 31-15 Reverse the direction of the Global Copy between local to remote
dscli> failoverpprc -dev IBM.2107-1301651 -remotedev IBM.2107-1300561 -type gcp -cascade
2000-2007:2000-2007
dscli>
2000-2007:2000-2007

dscli>
dscli>
Step 7: Start Metro Mirror from intermediate to local site

Metro Mirror is now established in the direction from intermediate to the local site with
Incremental Resync and force options and then again with Incremental Resync initialization.
The command to establish Metro Mirror is executed at the intermediate site. First it will be
established with a force option to recover and restart without doing a check at the local site for
any relationships. It will also take the change recording bitmaps that were created in “Step 2:
Start incremental re-synchronization at the intermediate site” on page 458 and merge it with
the out-of-sync bitmaps at the intermediate site. Then Metro Mirror is established with
Incremental Resync initialized, which will create new change recording bitmaps for Metro
Mirror at the intermediate site.
Note: At this point Metro Mirror will need to become full duplex and Global Copy will need
to complete a first pass before going on to the next step.
In Example 31-16 the Metro Mirror from intermediate to local is established with the option
-incrementalresync override, which causes the copying of all tracks recorded in the change
recording bitmap at the intermediate site. This can be monitored with the lspprc -l command
shown next in the example. When all tracks have been drained, the incremental
re-synchronization is enabled with the option -incrementalresync enable.
Example 31-16 Start Metro Mirror from intermediate to local with incremental re-synchronization
dscli> mkpprc -dev IBM.2107-1301261 -remotedev IBM.2107-1301651 -type mmir -mode nocp -incrementalresync override 2000-2007:2000-2007
dsclI>
lspprc -dev IBM.2107-1301651 -l 2000-2007
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs) Critical
Mode First Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG
=====================================================================================================================================================
======================================================================
2000:2000 Target Full Duplex - Metro Mirror 0 Disabled Invalid Enabled - 20 unknown Disabled
Invalid Disabled Disabled Disabled Disabled
2000:2000 Copy Pending - Global Copy 9 Disabled Enabled Invalid - 20 unknown Disabled
True Disabled Disabled Disabled Disabled
dsclI>
#
# Wait until all Out Of Sync Tracks has drained
#
dsclI>
=====================================================================================================================================================
===============================================================
dscli>
dscli>
Step 8: Start Global Mirror at local site

Once Metro Mirror is full duplex, Global Mirror can be started at the local site. If not already
done, the Global Mirror A session must be created and populated with the Global Copy
primary volumes at the local site. When the Global Mirror is started at the local site,
consistency groups will be formed and the FlashCopy targets at the remote site will refresh.
Note: In some cases, production may remain running at the intermediate site with a
three-site Metro/Global Mirror disaster recovery solution in place. In this case, no further
steps are taken.
In other cases, it may be necessary to restore production at the local site. In that case,
proceed to 31.3.3, “Returning to the original configuration” on page 463.
31.3.3 Returning to the original configuration

This section discusses the steps for moving production back to the local site from the
intermediate site and then restoring the original Incremental Resync Metro/Global Mirror
environment.

Figure 31-7 illustrates the steps to move Incremental Resync Metro/Global Mirror from the
intermediate site to the local site.
Primary
production
host Local Site
5
Remote Site
9
7
8 A
4
2 12
C
Intermediate Site 11
10
6 D
´Secondary 3
production
host B
1
13
Figure 31-7 Move incremental Resync Metro/Global Mirror back to local site
The following steps move production back to the local site and restore Incremental Resync
Metro/Global Mirror to its original configuration:
1. Stop applications at the intermediate site.
2. Suspend Metro Mirror from the intermediate site to the local site.
3. Fail over Global Copy from the remote to the intermediate site.
4. Terminate Metro Mirror at the local site.
5. Start applications at the local site.
6. Fail back Global Copy remote to the intermediate site.
7. Start Incremental Resync at the local site.
8. Terminate Global Mirror at the local site.
9. Suspend and terminate Global Copy from the local to the remote site.
10.Suspend Global Copy from the remote to the intermediate site.
11.Reverse Global Copy to run from the intermediate to the remote site.
12.Establish Metro Mirror at the local to the remote site.
13.Start Global Mirror at the intermediate site.
Step 1: Stop applications at the intermediate site

To prepare moving production back to the local site, all applications that are currently running
at the intermediate site are stopped.
Step 2: Suspend Metro Mirror from intermediate to remote

Before moving back to the local site, Metro Mirror is suspended from the intermediate to the
local site. This stops data from being copied to the local site.
Example 31-17 shows the DSCLI command to suspend the Metro Mirror.
Example 31-17 Suspend Metro Mirror from intermediate to remote

dscli>
dscli>
Step 3: Fail over Global Copy from remote to intermediate site

Next a failover with force and cascading options is issued from the remote to the intermediate
site. The force will bypass validation at the remote to determine if the remote is a secondary
of the intermediate, thus allowing the failover to be successful. The effect of the failover is to
change the state of the local site devices to suspended primary.
Example 31-18 shows the DSCLI command.
Example 31-18 Fail over Global Copy from remote to intermediate site
dscli> failoverpprc -dev IBM.2107-1300561 -remotedev IBM.2107-1301261 -type gcp -cascade -force
2000-2007:2000-2007
dscli>
dscli>
Step 4: Terminate Metro Mirror at local

In “Step 7: Start Metro Mirror from intermediate to local site” on page 462 the volumes at the
local site are Metro Mirror secondary. In the next the applications should be started at the
local site, which is not possible as long as these volumes are secondary volumes of the Metro
Mirror. Since the Metro Mirror was created with incremental resync enabled, a failover of the
Metro Mirror to the local site would discard the change recording bitmap at the intermediate
site. To avoid this the Metro Mirror is now terminated only at the local site. This clears the local
site from having any knowledge of being a secondary to the intermediate site.
Example 31-19 Terminate Metro Mirror at local


dscli>
dscli>
Step 5: Start applications at local site

Production can now be moved to the local site by restarting applications at the local site.
Step 6: Fail back remote to intermediate

With production now running at the local site and Global Copy still running from the local to
the remote site, the next steps are to prepare for reinstating Global Copy at the intermediate
site. A failback with force is executed from the remote to the intermediate site. The local site’s
out-of-sync bitmaps can be obtained for resync changes that may have occurred since the
swap. The remote site’s out-of sync bitmaps will contain changes made after the move back
to the local site. The intermediate site’s incremental resync change recording bitmaps will be
released during the failback. There may still be changes on the remote site that have not
made it to the intermediate site and still need to be updated.
Note: The writes from the remote to the intermediate site must be completely drained or
have completed the first pass before continuing to the next step. All changes at the local
site should have already been updated to the remote site, which is also being updated to
the intermediate site with the failover with force.
Example 31-20 shows how to issue the failbackpprc command with -force option. To monitor
the Out Of Sync Tracks a lspprc -l command must be issued at the remote site. The output
shows that the Global Copy from remote to intermediate is still a cascaded relation of the
Global Copy from local to remote.
Example 31-20 Fail back remote to intermediate

dscli> failbackpprc -dev IBM.2107-1300561 -remotedev IBM.2107-1301261 -type gcp -force 2000-2007:2000-2007
dscli>
ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs) Critical
Mode First Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG
=====================================================================================================================================================
======================================================================
2000:2000 Target Copy Pending - Global Copy 0 Disabled Invalid Enabled - 20 unknown Disabled
dscli>
dscli>lspprc -dev IBM.2107-1300561 -fullid 2000-2007
=====================================================================================================================================================
IBM.2107-1300561/2000:IBM.2107-1301261/2000 Copy Pending - Global Copy IBM.2107-1300561/20 unknown Disabled True
dscli>
dscli>
Step 7: Start incremental re-synchronization at local site

Incremental re-synchronization is established with Global Copy for the local to remote site
relationship. The incremental resync command will be enabled without initialization of the
changes bitmap (enablenoinit option). This will track the changes made to the local site so
that the intermediate can be re-synchronized in a later step after the remote is no longer
getting updates from the local site.
Example 31-21 shows the DSCLI to start incremental re-synchronization at local.
Example 31-21 Start incremental resync at local site

dscli> mkpprc -dev IBM.2107-1301651 -remotedev IBM.2107-1300561 -type gcp -mode nocp -cascade
-incrementalresync enablenoinit 2000-2007:2000-2007
dscli>
dscli>
Step 8: Terminate Global Mirror at local site

To begin the process of moving to a Metro/Global Mirror environment with production running
at the local site, Global Mirror is terminated. Global Mirror is currently running from the local to
the remote site, with FlashCopy targets being updated at the remote site. After this
relationship is terminated, the consistency groups at the remote FlashCopy targets will begin
to age.
Example 31-22 Terminate Global Mirror at local


dscli>
dscli>
Step 9: Suspend and remove Global Copy local to remote site

Now that Global Mirror (local to remote) has been terminated, Global Copy running from the
local to the remote site can be suspended and terminated. By suspending Global Copy, data
will stop being copied to the remote site. This will allow re-synchronization to complete from
the remote to the intermediate site. Once the re-synchronization of the intermediate is
complete, Global Copy from the local to the remote site can be terminated at the remote site.
By terminating only at the remote site, the status at the remote site will no longer be a Global
Copy secondary, which will allow a failback from intermediate to remote in a later step. In
addition, the local site will continue to have out-of-sync bitmaps in operation with its status
being suspended primary.
In Example 31-23 the Global Copy from the local to the remote site is suspended.
Subsequently, the Global Copy from remote to intermediate is queried to check that all Out Of
Sync Tracks has been drained. If this is the case the Global Copy relation at the remote site is
removed.
Example 31-23 Suspend and remove Global Copy local to remote at remote
dscli>
#
# Wait until all OOS has drained from remote to intermediate
#
dscli> lspprc -dev IBM.2107-1300561 -remotedev IBM.2107-1301261 -l -fmt default 2000-2007

ID State Reason Type Out Of Sync Tracks Tgt Read Src Cascade Tgt Cascade Date Suspended SourceLSS Timeout (secs) Crit
ical Mode First Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG
=====================================================================================================================================================
==========================================================================
2000:2000 Target Suspended Update Target Global Copy 0 Disabled Invalid Enabled - 20 unknown Disa
bled Invalid Disabled Disabled Disabled Disabled
2000:2000 Copy Pending - Global Copy 0 Disabled Enabled Invalid - 20 unknown Disa
bled True Disabled Disabled Disabled Disabled
dscli>
dscli> rmpprc -quiet -dev IBM.2107-1300561 -remotedev IBM.2107-1301651 -unconditional -at tgt 2000-2007
dscli>
dscli>
Step 10: Suspend Global Copy remote to intermediate

The process of reversing the Global Copy running from the remote to the intermediate is
initiated by first suspending the Global Copy relationship from remote to intermediate. This
should be done once the remote to intermediate site out-of-sync bitmaps is completely
drained from the last of the updates at the remote site, since Global Copy from the local site
has already been suspended.
Example 31-24 Suspend Global Copy remote to intermediated

dscli>
dscli>
Step 11: Reverse Global Copy to run intermediate to remote

To reverse the Global Copy relationship a failover and then failback must be done before
creating the Global Copy in the reverse direction.
A failover of Global Copy is first executed with cascading allowed from intermediate to remote
site. The failover will cause the intermediate site volumes status to be primary suspended and
sets up for local to intermediate and intermediate to remote connection. Next the failback
Global Copy intermediate to remote site is executed at the intermediate site with cascading
allowed and mode Global Copy.
The result of the failover and failback has successfully reversed the Global Copy relationship
to run from the intermediate to the remote site. Global Copy can now be established from the
intermediate to the remote site with Global Copy mode and cascading options.
Example 31-25 Reverse Global Copy to sun intermediate to remote

2000-2007:2000-2007
2000-2007:2000-2007

dscli>
dscli>
Step 12: Establish Metro Mirror from local to intermediate site

Metro Mirror is now established in the direction from local to intermediate site with
incremental resync and force and then again with incremental resync initialization. The
command to establish Metro Mirror is executed at the local site.
The forced mirror also causes the change recording bitmaps that were created in “Step 7:
Start incremental re-synchronization at local site” on page 467 to be merged with the
out-of-sync bitmaps at the local site. Then Metro Mirror is established with incremental resync
initialized, which will create new change recording bitmaps for Metro Mirror at the local site.
Note: At this point Metro Mirror will need to become full duplex and Global Copy will need
to complete first pass before going on to the next step.
Tip: In case that at the failure of the local site (see 31.3.1, “Local site fails” on page 455) a
freezepprc command has been issued, the paths from local to intermediate have been
removed. In order to succeed with the failback from local to intermediate the paths must be
reestablished right now.
Example 31-26 shows that the option -incrementalresync override is used first to copy the
track marked in the change recording bitmap at the local site. All Out Of Sync Tracks have
been drained to the intermediate site. When done, the incremental re-synchronization is
started with empty bitmaps at the local site.
Example 31-26 Establish Metro Mirror from local to intermediate site

dscli> mkpprc -dev IBM.2107-1301651 -remotedev IBM.2107-1301261 -type mmir -incrementalresync override 2000-2007:2000-2007
dscli>
dsclI>
#
# Wait until all OOS has drained
#
=====================================================================================================================================================
==============================================================
nvalid Disabled Disabled Disabled Enabled
dscli>
dscli>
=====================================================================================================================================================
==============================================================
nvalid Enabled Disabled Disabled Enabled
dscli>
dscli>
Step 13: Start Global Mirror

Once Metro Mirror is full duplex and Global Copy has completed a first pass, Global Mirror
can be started at the intermediate site and consistency groups will start forming successfully.
By starting Global Mirror at the intermediate site, the FlashCopy targets at the remote site will
start to refresh.
Note: You can now verify that Consistency Groups are forming successfully.
In Example 31-27 we just add the volumes to the existing Global Mirror because in “Step 3:
Terminate Global Mirror and suspend Global Copy” on page 459 we remove the application
host’s volumes from the session instead of terminating the whole Global Mirror. The Global
Mirror is inspected if consistency formation is ongoing by two subsequent showgmir -metrics
commands to see whether the Total Successful CG Count is increasing.
Example 31-27 Add volumes to session and check Global Mirror

dscli> chsession -dev IBM.2107-1301261 -lss 20 -action add -volume 2000-2007 ad
CMUC00147I chsession: Session ad successfully modified.
dscli>
AllowCascading
========================================================================================================================
=====
20 AD CG In Progress 2000 Active Primary Copy Pending Secondary Full Duplex True Enable
dscli>
ID IBM.2107-1301261/20

Last Successful CG Form Time 11/09/2006 20:40:31 MST
First Failure Control Unit IBM.2107-1301261
First Failure LSS Not Available
First Failure Status Error
First Failure Reason Members in Incorrect State
First Failure Master State Drain in Progress
Last Failure Control Unit IBM.2107-1301261
Last Failure LSS 0x40
Last Failure Status Error
Last Failure Reason Global Mirror Consistency Cannot be Maintained
Last Failure Master State Global Mirror Start Increment in Progress
Previous Failure Control Unit IBM.2107-1301261
Previous Failure LSS 0x20
Previous Failure Status Error
Previous Failure Reason Session or Session Members not in Correct State
Previous Failure Master State Global Mirror Run in Progress
dscli>
ID IBM.2107-1301261/20
First Failure Control Unit IBM.2107-1301261
First Failure LSS Not Available
First Failure Status Error
First Failure Reason Members in Incorrect State
First Failure Master State Drain in Progress
Last Failure Control Unit IBM.2107-1301261
Last Failure LSS 0x40
Last Failure Status Error
Last Failure Reason Global Mirror Consistency Cannot be Maintained
Last Failure Master State Global Mirror Start Increment in Progress
Previous Failure Control Unit IBM.2107-1301261
Previous Failure LSS 0x20
Previous Failure Status Error
Previous Failure Reason Session or Session Members not in Correct State
Previous Failure Master State Global Mirror Run in Progress
dscli>
dscli>
31.4 Failure at intermediate site scenario

The failure at the intermediate site scenario is used when the intermediate site becomes
unavailable or is inaccessible to the Metro/Global Mirror session. This scenario is divided into
four parts:
򐂰 When a disaster occurs at the intermediate, a recovery is done so that Global Mirror is run
from the local site to the remote site.
򐂰 Once the intermediate site is available again for Metro/Global Mirror, it is cleaned up.
򐂰 When the intermediate is cleaned up, it is then re-synchronized with the data that was
being copied from the local site to the remote site.
򐂰 When the intermediate is fully re-synchronized, the original configuration can be restored
and Global Mirror is started at the intermediate site.
31.4.1 Intermediate site failure

This section describes the steps for recovery from failure at the intermediate site while
production continues at the local site.
When the storage at the intermediate site is no longer accessible (failure), data will be copied
from the local to the remote site using the incremental re-synchronization implementation in
place.
When transitioning from the original MGM configuration to the configuration of the local to the
remote site, consistency groups are formed at local and drained to remote, while data
consistency is retained between the two sites. Production will be able to continue at the local
site until the intermediate site is operational again. Figure 31-8 illustrates the steps to recover
from a failure at the intermediate site while production is able to continue at the local site.
Primary
production
host Local Site
Remote Site
5
A
6
1
C
Intermediate Site
2
D
4
Figure 31-8 Failure at the intermediate
The steps for recovery after failure at the intermediate site are:
1. Suspend Metro Mirror at local site.
2. Clean up surviving components of Global Mirror (if possible).
3. Fail over Global Copy at remote site.
4. Verify Global Mirror consistency group.
5. Start Global Copy from local to remote site.
6. Create sessions and restart Global Mirror at local site.
Step 1: Suspend Metro Mirror at local site

Metro Mirror at the local site may already be suspended due to the failure of the intermediate.
However, in the case where there is no I/O running, Metro Mirror will not suspend. In this

case, the pausepprc command can be used. Since the intermediate may not be accessible
due to the failure, the -unconditional -at src options will suspend the primary volumes and
ignore the target volumes, therefore allowing the suspension at the local site. This step
ensures that all remaining pairs are suspended at the local site.
Example 31-28 shows the command to suspend the Metro Mirror at the local site.
Example 31-28 Suspend Metro Mirror at local site

dscli> pausepprc -dev IBM.2107-1301651 -unconditional -at src 2000-2007
CMUC00157I pausepprc: Remote Mirror and Copy volume pair 2000: relationship successfully paused.
dscli>
dscli>
Step 2: Clean up surviving components of Global Mirror (if possible)

Depending on the type of failure at the intermediate, there may be some components of
Global Mirror that can be cleaned up. If the intermediate site is still partially accessible, a
terminate of Global Mirror can be attempted by executing the command rmgmir. This
command may not succeed depending on the nature the failure. If the command fails, it could
mean that the subordinates may be orphaned due to the master at the intermediate site not
having access to them. In this case, the rmgmir command can be attempted at any of the
subordinates. If none of the cases mentioned were successful and Global Mirror or any of its
components cannot be cleaned up, it will be done once the intermediate is accessible again
in 31.4.2, “Intermediate site back up” on page 477.
Note: Depending on the extent of the failure at the intermediate, Global Mirror may no
longer be running, and may show “FATAL” or failing consistency groups. If Global Mirror
was in the middle of a FlashCopy, then the consistency group may need to be verified at
some point. To check the status of the Global Mirror and see if consistency groups are
failing, the command showgmir -metrics will list the current status.
Step 3: Fail over Global Copy at remote site

Global Copy is failed over at the remote site with cascading allowed to change the state of the
volumes at the remote from secondary duplex pending (or suspended) to suspend host source.
The data will be registered in the out-of-sync bitmaps and will be used during the
re-synchronization of the intermediate in 31.4.3, “Re-synchronization at intermediate” on
page 480.
The command issued at the remote site is a failoverpprc with the -cascade option, as
Example 31-29 Fail over Global Copy at remote site

2000-2007:2000-2007
dscli>
dscli>
Step 4: Verify Global Mirror consistency group

It is essential to check the Global Mirror consistency at the remote site. This step is necessary
in case the intermediate site failed in the middle of a consistency group formation, which
would mean that the FlashCopy of the Global Mirror will need to be reverted or committed.
To verify the Global Mirror consistency, see 27.7, “Check consistency at the remote site” on
page 400 describes how to verify the consistency group and determine if any action needs to
be taken.
Tip: The lsflash -l command will show a query of the FlashCopy status. This will be
helpful when verifying the Global Mirror consistency group.
Step 5: Start Global Copy from local to remote site

The incremental resync with copy option is established from the local to the remote site by
issuing the command mkpprc with the option -incrementalresync recover. The recover
parameter for the option -incrementalresync will do a check to see if there was a former
relationship at the remote site. In “Step 2: Clean up surviving components of Global Mirror (if
possible)” on page 474 the local site was failed over and it is no longer in a relationship.
When the Global Copy with incremental resync is completely established, the function that
was running previously at the local site is stopped. The current change recording bitmaps at
the local site are left alone and merged with the out-of-sync bitmaps.
If there was a former incremental resync relationship at the remote site, then the override
parameter must be used with the -incrementalresync option when establishing Global Copy.
Important: All writes are transferred at this point from the local to the remote site and all
the out-of-sync tracks must have drained before continuing to the next step. To query
out-of-sync tracks, issue the command lspprc -l at the local site.
In Example 31-30 the lspprc command issued at the local site shows the relation of the
Metro Mirror from the local to the intermediate site. After the mkpprc with -incrementalresync
recover the pair relation has changed to the Global Copy relation local to remote.
Example 31-30 Start Global Copy from local to remote with incremental re-synchronization
================================================================================================================================================================
IBM.2107-1301651/2000:IBM.2107-1301261/2000 Suspended Internal Conditions Target Metro Mirror IBM.2107-1301651/20 300 Disabled Invalid
dscli>
dsclI> mkpprc -dev IBM.2107-1301651 -remotedev IBM.2107-1300561 -type gcp -mode full -incrementalresync recover 2000-2007:2000-2007

dsclI>
dsclI> lspprc -dev IBM.2107-1301651 -l -fullid 2000-2007
Timeout (secs) Critical Mode First Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG
=============================================================================================================================================================
===================================================================================================
IBM.2107-1301651/2000:IBM.2107-1300561/2000 Copy Pending - Global Copy 9 Disabled Disabled Invalid - IBM.2107-13016
51/20 300 Disabled True Disabled Disabled Disabled
Disabled
Disabled
Disabled
Disabled
Disabled
Disabled
Disabled
Disabled
dscli>
dscli>
Step 6: Create session and start Global Mirror at local site

For the Global Copy relation from the local site to the remote site, we create a session and
add volumes to the session prior to starting the new Global Mirror at the local site. The
session is created using the mksession command, and the volumes can then be added to the
session by issuing the chsession command with the option -action add at the local site.
The Global Mirror session can now be started by issuing the mkgmir command at the local
site. This configuration remains unchanged until the intermediate site is available again for
Metro/Global Mirror.
Production continues to run at the local site without interruption while the original
Metro/Global Mirror configuration of local to intermediate to remote site transitions to local to
remote site.
Example 31-31 shows the steps to create sessions and Global Mirror and how to check it.
Example 31-31 Create session and start Global Mirror at local site
dscli> mksession -dev IBM.2107-1301651 -lss 20 10
dscli>
dscli> chsession -dev IBM.2107-1301651 -lss 20 -action add -volume 2000-2007 10
dsclI>
dsciI> lssession -dev IBM.2107-1301651 20 10
=================================================================================================================
dscli>
dscli> mkgmir -dev IBM.2107-1301651 -lss 20 -session 10
dscli>
ID IBM.2107-1301651/20
First Failure LSS -
Last Failure LSS -
dscli>
dscli> lssession -dev IBM.2107-1301651 20 10
=========================================================================================================================
dscli>
dscli>
31.4.2 Intermediate site back up

In this section we describe the process of cleaning up the intermediate site once it becomes
available again. The cleanup is required before re-synchronizing the intermediate, as
discussed in 31.4.3, “Re-synchronization at intermediate” on page 480.
There may be Metro Mirror or Global Mirror relationships or both that need to be terminated.
These relationships, depending on how the intermediate was lost, may not have been
cleaned up prior to the failure of the intermediate in Section “Step 2: Clean up surviving
components of Global Mirror (if possible)” on page 474. The components that were not
terminated at that stage must be terminated now.

Figure 31-9 illustrates the steps to clean up the intermediate once it is recovered.
Local Site
Remote Site
Primary
A
production
host
C
Intermediate Site
1 2 3
Figure 31-9 Cleanup of the intermediate when it becomes available
The steps to cleanup the intermediate after it is fully recovered are as follows:
1. Remove Metro Mirror.
2. Suspend Global Copy intermediate to remote at intermediate.
3. Terminate former Global Mirror.
Step 1: Remove Metro Mirror

Metro Mirror, previously running from the local to the intermediate site, will need be
terminated at the intermediate site. When the intermediate is available, the volumes may still
show as Target Full Duplex from the former Metro Mirror relationship. Removing the Metro
Mirror will allow for the failback from the remote to the intermediate site to be done in a later
step.
The command rmpprc issued at the intermediate site will terminate all Metro Mirror
relationships at the intermediate.
Tip: To remove the Metro Mirror pairs, the communication between local and intermediate
site must work. We recommend that you check the PPRC paths in both directions when the
intermediate site became available again.
In Example 31-32 the paths between local and intermediate are checked first. When the
Metro Mirror was removed at the local site the Global Copy between local and remote is show
with the subsequent lssprc command issued at the local site.
Example 31-32 Check paths and remove Metro Mirror

dcsli> lspprcpath -fmt default -fullid -dev IBM.2107-1301651 20
===================================================================================================================
IBM.2107-1301651/20 IBM.2107-1301261/20 Success FF20 IBM.2107-1301651/I0003 IBM.2107-1301261/I0200 5005076303FFC04D
IBM.2107-1301651/20 IBM.2107-1301261/20 Success FF20 IBM.2107-1301651/I0200 IBM.2107-1301261/I0330 5005076303FFC04D
IBM.2107-1301651/20 IBM.2107-1300561/20 Success FF20 IBM.2107-1301651/I0000 IBM.2107-1300561/I0200 5005076303FFC02C
IBM.2107-1301651/20 IBM.2107-1300561/20 Success FF20 IBM.2107-1301651/I0003 IBM.2107-1300561/I0003 5005076303FFC02C
dscli>
dscli> rmpprc -quiet -dev IBM.2107-1301261 -remotedev IBM.2107-1301651 -unconditional -at tgt 2000-2007
dscli>
dscli> lspprc -dev IBM.2107-1301651 -remotedev IBM.2107-1301261 -fullid -fmt default 2000-2007
Status
==========================================================================================================================================
====
dscli>
dscli>
Step 2: Suspend Global Copy intermediate to remote at intermediate

The volumes at the intermediate site may still have Global Copy relationships with the remote
site. In some cases, these Global Copy relations may already be suspended due either to the
failure of the intermediate or from the steps taken in “Step 2: Clean up surviving components
of Global Mirror (if possible)” on page 474.
The command pausepprc issued at the intermediate site will suspend the Global Copy
relations between the intermediate and the remote site. The -unconditional -at src option must
be used with the pausepprc command, because Global Copy needs only to be suspended at
the intermediate, the Global Copy primary. The previous Global Copy secondary at the
remote site is already in a failover state, and thus is primary suspended.
Example 31-33 shows how to suspend Global Copy at the intermediate site.
Example 31-33 Suspend Global Copy intermediate to remote

dscli>
dscli>

Step 3: Terminate former Global Mirror
The former Global Mirror session will also need to be terminated at the intermediate site. The
command rmgmir will terminate Global Mirror when issued at the intermediate site. However,
if this operation was already successfully executed at “Step 2: Clean up surviving
components of Global Mirror (if possible)” on page 474, it will fail if attempted again because
the Global Mirror session was already terminated.
It if was not terminated before and fails again now, it could be caused by orphaned
subordinates. In this case, Global Mirror must be terminated at the orphaned subordinate by
using the command rmgmir.
Note: The failover state at the remote site will prevent any previous Global Mirror
configuration from operating.
31.4.3 Re-synchronization at intermediate

Once the cleanup of the intermediate is complete, re-synchronization at the intermediate can
begin. During the re-synchronization, the data will be copied from the remote to the
intermediate.
Figure 31-10 illustrates the steps to re-synchronize the intermediate before restoring the
original configuration.
Primary
production
host Local Site
Remote Site
2
A
C
Intermediate Site
D
1
Figure 31-10 Re-synchronization of the intermediate
The steps to re-synchronize the intermediate are as follows:

1. Fail back Global Copy remote to intermediate site.
2. Start incremental re-synchronization at local site.
Step 1: Fail back Global Copy remote to intermediate site
At “Step 2: Clean up surviving components of Global Mirror (if possible)” on page 474, the
remote site volumes were changed to primary volumes suspended in preparation for the
failback to the remote site.
We can now issue a failbackpprc command at the remote site, and this begins copying data
from the remote site to the intermediate site.
Note: Waiting for the initial pass of the re-synchronization before restarting incremental
re-synchronization is good practice to reduce the number of updates sent later when Metro
Mirror is started with incremental re-synchronization and force at the local site. To query
the out-of-sync status, the lspprc -l command is issued at the intermediate site.
Example 31-34 shows the command to fail back the Global Copy.
Example 31-34 Fail back Global Copy remote to intermediate site

dscli> failbackpprc -dev IBM.2107-1300561 -remotedev IBM.2107-1301261 -type gcp 2000-2007:2000-2007
dscli>
dscli>
Step 2: Start Incremental Resync at local site

Incremental re-synchronization can now be started at the local site with the no initialization
option.
The command mkpprc with the option -incrementalresync enablenoinit will successfully start
Incremental Resync at the local site without enabling the bitmaps.
Note: This step is necessary to allow the Metro Mirror relationship at the local site to be
restored in a later step using the -incrementalresync override option.
Example 31-35 Start incremental resync at local site

dscli> mkpprc -dev IBM.2107-1301651 -remotedev IBM.2107-1300561 -type gcp -mode nocp -incrementalresync
enablenoinit 2000-2007:2000-2007
dscli>
dscli>

31.4.4 Restore original configuration
Once re-synchronization at the intermediate has completed, the original configuration can be
restored without interrupting the production at the local site. Figure 31-11 illustrates the steps.
Primary
production Local Site
host
Remote Site
2
A 3
1
C
Intermediate Site 5
D
4
7 B
Figure 31-11 Restoring the Metro/Global Mirror configuration
The steps to restore the original configuration are as follows:

1. Stop Global Mirror at the local site.
2. Suspend Global Copy at local to remote site.
3. Stop Global Copy local to remote at the remote site.
4. Suspend and fail over Global Copy from the remote to intermediate site.
5. Fail back Global Copy at the intermediate to remote site.
6. Create Metro Mirror with Incremental Resync at the local site.
Step 1: Stop Global Mirror at local site

Global Mirror has been running from the local to remote site while the intermediate was being
recovered. Before restoring the original configuration, the Global Mirror with Incremental
Resync from the local site to the remote site is terminated. The command rmgmir is issued at
the local site. As a result, the remote FlashCopy target will begin to age while the transition
back to the original configuration is in progress.
Note: The swap back to the intermediate can be done at any time but would normally be
done at a planned time.
Example 31-36 Stop Global Mirror at local site
dscli>
dscli>
Step 2: Suspend Global Copy at local to remote

To stop data being copied to the remote site and to allow the re-synchronization to complete
between the remote and intermediate sites, Global Copy at the local site is suspended. The
pausepprc command is issued at the local site, which will primary suspend the local and
primary suspend the remote.
Important: Out-of-sync tracks need to be drained completely to the remote. Wait until this
is the case before continuing to the next step. To query the out-of-sync tracks, the lspprc
-l command can be issued from the remote site.
Example 31-37 shows the DSCLI command to suspend the Global Copy.
Example 31-37 Suspend Global Copy at local to remote

dscli>
dscli>
Step 3: Stop Global Copy local to remote at remote site

For the remote to lose knowledge of being a secondary of the local, the Global Copy is only
terminated at the remote site. The command rmpprc with the -unconditional -at tgt option is
issued at the remote site. This termination does not affect the local site’s state and therefore
allows for the out-of-sync bitmaps to remain in operation at the local site. The state of the
local will remain primary suspended while the remote will no longer show as suspended
target.
This step is necessary to allow the fail back of the intermediate to remote site in a later step.
Note: The local site has updates for the intermediate and remote being recorded in the
incremental resync change recording and out-of-sync bitmap.

Example 31-38 shows the command to remove the Global Copy to the remote site. The
lspprc command, which was issued at the local site, shows that the relation at the local site is
still there. It is still required for the incremental re-synchronization in “Step 6: Create Metro
Mirror with incremental resync at local site” on page 485.
Example 31-38 Stop Global Copy local to remote at remote site

dscli>
#
# At Local Site
#
dscli>lspprc -dev IBM.2107-1301651 -remotedev IBM.2107-1300561 -fullid -fmt default 2000-2007

Status
==========================================================================================================================================
======
dscli>
dscli>
Step 4: Fail over Global Copy remote to intermediate site

In this step the Global Copy between the intermediate and the remote site is prepared to be
reversed into the initial direction from intermediate to remote.
The command issued is failoverpprc at the intermediate. Because the volumes at the
intermediate are cascaded, the -cascade option is issued with the failoverpprc command.
This will make the intermediate primary suspended.
Note: In “Step 2: Suspend Global Copy at local to remote” on page 483 Global Copy was
suspended at the local, thus stopping any new data from going to the remote. At this point,
out-of-sync tracks should be drained. To check the out-of-sync tracks, issue the lspprc -l
command at the remote site.
Example 31-39 shows the appropriate DSCLI command to fail over the Global Copy.
Example 31-39 Fail over Global Copy remote to intermediate site

2000-2007:2000-2007
dscli>
dscli>
Step 5: Fail back Global Copy at intermediate to remote site

The failbackpprc command is issued at the intermediate site to fail back Global Copy at the
intermediate site. Since Global Copy is in a cascaded relation, the failbackpprc would be
issued with the copy type gmir with the -cascade option. This starts Global Copy from the
intermediate to the remote site.
Example 31-40 Fail back Global Copy at intermediate to remote site

2000-2007:2000-2007
dscli>
dsclI>
Step 6: Create Metro Mirror with incremental resync at local site

The command to create Metro Mirror with the incremental resync option is issued twice in this
step using two different parameters. First, to stop incremental resync from the local to the
remote site and to be able to move it to the intermediate site, Metro Mirror with incremental
resync is first set by issuing the mkpprc command with the -incrementalresync override
option. By using the override parameter for -incrementalresync, Metro Mirror with incremental
resync will be established without doing a check at the intermediate site (the Metro Mirror
secondary). The change recording bitmaps are also merged with the out-of-sync bitmaps at
the local site during this step.
Next, to monitor and track data as it is being written on the primary volumes at the local site, a
Metro Mirror relationship with incremental resync is created from the local to the intermediate
site by using the mkpprc command with the -incrementalresync enable option. By using the
enable parameter for -incrementalresync, incremental resync is initialized by creating a new
change recording bitmap on the local site.
Note: Both Metro Mirror and Global Copy may still be in first pass. To query the status of
Global Mirror or Metro Mirror, use the lspprc -l command at the local site to query Metro
Mirror and at the intermediate site to query Global Copy.
Example 31-41 shows the DSCLI command to create the Metro Mirror with -incremental
override. During this phase all tracks marked in the incremental re-synchronization are not
enabled. When all tracks have been drained to the intermediate site the incremental
re-synchronization is enabled.
Example 31-41 Create Metro Mirror with incremental resync at local site
dscli> mkpprc -dev IBM.2107-1301651 -remotedev IBM.2107-1301261 -type mmir -mode nocp -incrementalresync override 2000-2007:2000-2007

dscli>
==========================================================================================================================================
=========================================================================
Disabled Invalid Disabled Disabled Disabled Enabled
dscli>
dscli>mkpprc -dev IBM.2107-1301651 -remotedev IBM.2107-1301261 -type mmir -mode nocp -incrementalresync enable 2000-2007:2000-2007
dscli>
==========================================================================================================================================
=========================================================================
dscli>
dscli>
Step 7: Start Global Mirror at intermediate site

Once the original configuration is restored and Metro Mirror is full duplex, then Global Mirror
can be started and it will start forming consistency groups successfully. Global Mirror is
started by issuing the mkgmir command from the intermediate. Depending on the Global
Mirror configuration shown after the intermediate site has become available again, it is
possible that the sessions also have to be created.
Important: Verify that the out-of-sync tracks have completed for Metro Mirror before
starting Global Mirror. Otherwise, the consistency groups will start to fail when Global
Mirror is started with the mkgmir command.
Example 31-42 shows how to start the Global Mirror and the sessions at the intermediate
site.
Example 31-42 Start Global Mirror at intermediate site

dscli> mksession -dev IBM.2107-1301261 -lss 20 ad
CMUC00145I mksession: Session AD opened successfully.
dscli>
dscli> chsession -dev IBM.2107-1301261 -lss 20 -action add -volume 2000-2007 ad
CMUC00147I chsession: Session AD successfully modified.
dscli>
AllowCascading
========================================================================================================================
=====
dscli>
dscli> mkgmir -dev IBM.2107-1301261 -lss 20 -session ad
CMUC00162I mkgmir: Global Mirror for session ADsuccessfully started.
dscli>
dscli>

Part 8
Part 8 Interoperability
In this part we discuss the interoperability of the various Copy Services functions on the
DS8000, and also the interoperability of the DS8000 with other IBM System Storage and
TotalStorage disk subsystems in Copy Services implementations.

32
Chapter 32. Data migration through double

cascading
In this chapter we discuss how to ensure a consistent data migration by combinig copy
services functions in a double cascading configuration.

32.1 Data migration with double cascading
Combining the Copy Service functions Metro Mirror and Global Mirror into an environment
with double cascading allows us to maintain data consistency while migrating data.
There are two possibilities for accomplishing data consistency during a migration using an
environment that has double cascading. The first possibility is to shut down all applications at
the local site and let the out-of-sync drain completely. The second possibility is to change the
Global Copy relationships to Metro Mirror, then when the out-of-sync is approaching zero, a
freeze is issued to the changed relationships.
DWDM DWDM
A PPRC-MM B PPRC-GC PPRC-GC C PPRC-GC D

DS8K DS8K <50 Kms DS8K DS8k
Local Secondary Tertiary Remote

Figure 32-1 Double cascading example
Figure 32-1 shows double cascading with a Metro Mirror relationship from the local to the
secondary and Global Copy from the secondary to the tertiary site and from the tertiary to the
remote site. There is a cascading relationship at both the secondary and tertiary sites where
the volumes are both secondaries and primaries. In this example, if all applications are
stopped at the local site, then the local, secondary, tertiary and remote will reach a point after
some time where they are all equal. Therefore they will be consistent and data has
successfully been migrated to the remote site volumes.
The second approach to provide data consistency during migration, again looking at the
example in Figure 32-1, a freeze is first issued to the Metro Mirror relationship from the local
to the secondary site. Since Metro Mirror is running between the local and the secondary site,
there is already a synchronous relationship. Therefore the secondary volumes are consistent
with the local site volumes at this time. The next step would be to change all the Global Copy
relationships to Metro Mirror relations and then issue a freeze when the out-of-sync has
reached zero. When the out-of-sync is fully drained, then there is a consistent relationship
from the secondary to the remote and migration to the remote is complete. Data migration
has sent a snapshot of data to the remote site so that there is consistency. This process
would be used when production cannot be stopped, since production remains to be running
at the local site. However, once the freeze is first issued on the Metro Mirror relationship, from
that point forward the remote will only be consistent with the secondary. The local during this
process is still being updated and is changing. The remote will have a snapshot of the time
since the freeze was issued.
DWDM DWDM
A PPRC-- GC B PPRC-- GC PPRC-- GC C PPRC-- MM D

DS8K DS8K <50 Kms DS8K DS8k
Local Secondary Tertiary Remote

Figure 32-2 Global Copy changed to Metro Mirror and direction reversed
Once data migration and consistency have been accomplished at the remote site, you can
start production at the remote site, if desired. In this case all applications must first be
stopped at the local site and the direction of the mirror is reversed so that Metro Mirror is now
running from the remote to the tertiary site and Global Copy is running from the tertiary site to
the secondary site and then to the primary site. Figure 32-2 shows the new configuration with
the Metro Mirror and Global Copy reversed and production running at the remote site.
In this example, it is shown that to protect the production at the remote, the remote copy
relations can be reversed to use the tertiary, secondary, or local volumes as targets. These
copy relations would be created before starting production at the remote site. Since the
relationships from the secondary to the tertiary and from the tertiary to the remote are both in
a cascaded relationship, to reverse the directions Global Copy must be first removed and
then recreated in the reverse directions.
Note: No I/O should be running at any time at any of the sites while the reversal of the
remote copy relations is being done. This would corrupt the data resulting and
consequently requiring a full copy.
Chapter 32. Data migration through double cascading 493

33
Chapter 33. Interoperability between DS6000

and DS8000
In this chapter we show the interoperability between the Copy Services functions in the
DS6000 and the DS8000. This chapter contains the following sections:
򐂰 DS8000 and DS6000 Copy Services interoperability
򐂰 Preparing the environment
򐂰 RMC: Establishing paths between DS6000 and DS8000
򐂰 Managing Metro Mirror or Global Copy pairs
򐂰 Managing DS6000 to DS8000 Global Mirror
򐂰 Managing DS6000 and DS8000 FlashCopy
Note: Remote Mirror and Copy (RMC) was formerly called Peer-to-Peer Remote Copy
(PPRC). All references to PPRC are interchangeable with RMC.

33.1 DS8000 and DS6000 Copy Services interoperability
Copy Services operations are supported between the DS8000 and the DS6000. This means
you can have a mixed Copy Services environment that contains both devices.
33.2 Preparing the environment

Before starting Copy Services operations in a mixed DS6000/DS8000 environment, you need
to ensure this environment is set up correctly.
33.2.1 Minimum microcode levels

To manage Copy Services in a mixed environment, you need to have licensed internal code
bundle 6.0.600.9 or later on your DS6000. You also need your IBM Service Representative to
install code bundle 6.0.500.52 or later on your DS8000.
33.2.2 Hardware and licensing requirements

To establish Remote Mirror and Copy pairs (also known as PPRC pairs) between a DS6000
and a DS8000, the DS6000 and the DS8000 must both have the appropriate Remote Mirror
and Copy (RMC) licenses.
The DS6000 and DS8000 must also have Fibre Channel adapters that have connectivity to
each other, either directly or via Fibre Channel switches. ESCON adapters cannot be used to
mirror a DS6000 with a DS8000.
33.2.3 Network connectivity

Network connectivity requirements depend on how you want to manage the environment.
For FlashCopy management:

򐂰 If you want to use the DS8000 DS GUI to manage FlashCopy on the DS6000, then you
need network connectivity between the DS8000 HMCs and the DS6000 SMCs.
򐂰 If you want to use the DS6000 DS GUI to manage FlashCopy on the DS8000, then you
need network connectivity between the DS8000 HMCs and the DS6000 SMCs.
򐂰 If you want to use the DS CLI to manage FlashCopy on either the DS6000 or the DS8000
then the DS8000 HMCs and the DS6000 SMCs do not need to be connected via a
network. However, the machine on which the DS CLI is installed needs to have network
connectivity to both the DS SMC and the DS HMC.
򐂰 If you have PPRC links between the DS6000 and DS8000, you can also use remote
FlashCopy to establish FlashCopies of the Remote Mirror target volumes.
For creating Remote Mirror and Copy (RMC), also known as Peer-to-Peer Remote Copy
(PPRC) paths and pairs:
򐂰 In a mixed environment, one machine will be the source machine (local) and one will be
the target machine (remote). If you want the ability to manage the RMC paths and pairs
from either machine, then you will need network connectivity between the DS6000 SMC
and the DS8000 HMC. If you use the DS CLI, the DS SMC and DS HMC do not need to
be able to communicate with each other, but the DS CLI machine needs to be able to
communicate with both the DS HMC and the DS SMC.
򐂰 If you do not plan to use the remote system as a source and you do not intend to use the
DS Storage Manager GUI to manage the pairs and paths, then network connectivity to the
remote machine (assuming the remote machine is at a remote site) is not necessary. This
is because, when you use the DS CLI, all path and pair establishment is done by
connecting to the source machine (the local machine). In a disaster recovery scenario,
you will have to travel to the remote site to perform management tasks for the remote
machine. This setup is not recommended because it is less flexible.
33.2.4 Create matching user IDs and passwords

When you want to use the DS CLI or DS GUI to perform Copy Services operations, you must
authenticate with a valid user ID and password. When you use the DS6000 DS GUI to
perform an operation that requires it to issue a command to a DS8000, it must authenticate
with the DS8000. The DS user ID and password that you used to log on to the DS6000 DS
GUI is used to authenticate with the DS8000 HMC. The same applies if you use the DS8000
DS GUI to manage a DS6000. This means that the same user ID and password must be
defined in both the SMC and the HMC. This task must be manually performed. If, instead of
the DS GUI, you only use the DS CLI, then this requirement does not exist. However, it is still
of benefit to use a matching user ID and password to manage machines in either Storage
Complex.
Note: A DS8000 Storage Complex is made of up of either one or two HMCs that together
manage one or two DS8000 Storage Units. A DS6000 Storage Complex consists of either
one or two SMCs that together manage one or two DS6000 Storage Units.
33.2.5 Updating the DS CLI profile

If you plan to use the DS CLI to manage your mixed environment, you can create a profile to
simplify your connection to the DS CLI, commands, and script operations. To achieve this,
add extra lines, as shown in Example 33-1. In this example, the DS6000 is considered the
local system, while the DS8000 is considered the remote system.
Example 33-1 Possible modification to a DS CLI profile

# DS6000
# hmc1 is the DS6000 SMC
hmc1: 10.0.0.100
# devid is the serial number of the DS6000
devid: IBM.1750-1300247
# remotedevid is the serial number of the DS8000
remotedevid: IBM.2107-7503461
# Username is a user created on the ESS Specialist that matches the userid on the DS6000
username:admin
# The password for the admin user id. Placing it here is not very secure.
password:passw0rd
# The password file is created using the managepwfile and is a better way to manage this.
# pwfile:security.dat
Placing the password in the profile is not very secure (because it is stored in a plain text file),
but may prove more convenient. A password file can be created using the managepwfile and
is a better way to manage this. After creating the password file (which by default is called
security.dat), you can remove the password from the profile and instead specify the pwfile
file.
Chapter 33. Interoperability between DS6000 and DS8000 497

The command to create a passwd file in this example is:
managepwfile -action add -name admin -pw passw0rd
A simple method when you have multiple servers to manage is to create multiple profile files.
Then, when starting the DS CLI, you can specify the profile file you wish to use with the -cfg
parameter. In a Windows environment you could have multiple Windows batch files (BAT
files), one for each machine you wish to manage. The profile shown in Example 33-1 on
page 497 could be saved in the C:\program files\ibm\dscli\profile directory as
1750source.profile. Then you could create a simple Windows BAT file with three lines, as
Example 33-2 Windows BAT file to start a specific profile

title DS CLI Local DS6800 1300247 Remote IBM.2107-7503461
cd C:\Program Files\ibm\dscli\profile
dscli -cfg 1750source.profile
We save the BAT file onto the Windows desktop and start it by double-clicking the icon. The
DS CLI will open and will start the specified profile. By creating a BAT file and profile for each
machine, we can simplify the process of starting the DS CLI. We can also specify the profile
when starting the DS CLI from inside a script, or when running the DS CLI in script mode.
33.2.6 Adding the Storage Complex

If you wish to use a single DS GUI to manage both FlashCopy and Remote Mirror and Copy
pairs on your DS6000 and DS8000, you have to add the Storage Complex of one Storage
Unit (such as a DS6000) to the Storage Complex of the other Storage Unit (such as a
DS8000). Note that we have to do this operation once in each direction. In other words, you
have to add the DS6000 to the DS8000 and then add the DS8000 to the DS6000.
Adding the DS6000 Storage Complex to a DS8000 Storage Complex

You add a DS6000 Storage Complex to a DS8000 Storage Complex as follows.
the DS6000 SMC and that it has the same password. If not, the operation to add the
Storage Complex will fail. You will need to always use this same user ID for multi-complex
management.
1. Connect to the DS8000 HMC using the DS GUI.

5. Select Add.
Figure 33-1 shows the Add Storage Complex panel.
Figure 33-1 Add DS6000 complex to DS8000
6. Enter the IP address of the DS6000 SMC into the Management console 1 IP address box.
If you have a second SMC, select the Define a second Management console box and
enter the second SMC address into the Management console 2 IP address box. Now
select OK.
7. When the add Storage Complex operation completes, the Storage Complex screen will
now show the DS6000 SMC as an additional Storage Complex.
8. Having added the DS6000 Storage Complex to the DS8000 Storage Complex, you are
now able to use the DS8000 DS GUI to create paths and Remote Mirror and Copy pairs,
where the DS6000 is the source device. You can also use the DS8000 DS GUI to manage
FlashCopy pairs on the DS6000.
Note: The steps to add the DS6000 Storage Complex to the DS8000 Storage Complex
cannot be performed using the DS CLI. This is by design. However, if you do not plan to
use the GUI, this will not be an issue.
Adding the DS8000 Storage Complex to a DS6000 Storage Complex

You add a DS8000 Storage Complex to a DS6000 Storage Complex as follows.
the DS8000 HMC and that it has the same password. If not, the operation to add the
Storage Complex will fail. You will need to always use this same user ID for multi-complex
management.
1. Connect to the DS6000 SMC using the DS GUI.

5. Select Add Storage Complex.

Figure 33-2 shows the Add Storage Complex panel.
Figure 33-2 Add DS8000 complex to DS6000
6. Enter the IP address of the DS8000 HMC in the Management console 1 IPfield. If you
have a second DS8000 HMC, select Define a second Management console and enter
the second HMC address in the Management console 2 IP field. Click OK.
7. When the add Storage Complex operation completes, the Storage Complex panel shows
the DS8000 HMC as an additional Storage Complex. In Figure 33-3, a DS6000 SMC, an
ESS 800 Copy Services Server, and a DS8000 HMC are all defined in one DS6000 DS
GUI.
10.0.0.1
10.0.0.2
10.0.0.3
Figure 33-3 DS6000 GUI showing multiple complexes
8. Having added the DS8000 Storage Complex to the DS6000 Storage Complex, you are
now able to use the DS6000 DS GUI to create paths and Remote Mirror and Copy pairs,
where the DS8000 is the source device. You can also use the DS6000 DS GUI to manage
FlashCopy pairs on the DS8000.
Note: The steps to add the DS8000 Storage Complex to the DS6000 Storage Complex
cannot be performed using the DS CLI.
User ID management
There are two important considerations when managing multiple Storage Complexes from a
single DS GUI:
򐂰 The user ID you use to add the alternative complex must exist on both complexes. You
must continue to use this user ID for all multi complex operations. If you log on to either
DS GUI with a user ID that does not exist on the other complex, the remote complex will
not appear in the Storage Complex screen.
򐂰 Even when you have successfully added one complex to another, new user IDs created in
one complex will not mirror to the other complex. They have to be manually created and
managed in each complex.
Storage management
You cannot use the DS8000 DS GUI to perform storage configuration on a DS6000.
Likewise, you cannot use the DS6000 DS GUI to perform storage configuration on a DS8000.
You can only perform Copy Services management tasks on the alternative device. If you are
logged onto the DS6000 DS GUI and wish to configure some storage on the DS8000, you will
need to log off and then log on to the DS8000 DS GUI. The same applies if you are logged
onto the DS8000 DS GUI and wish to configure storage on the DS6000. You must log on to
the DS6000 DS GUI to do this.
33.2.7 Volume size considerations for Remote Mirror Copy

When RMC (PPRC) pairs are created, it is very important that the two volumes, source and
target, be exactly the same size. In RMC it is not an issue if the target volume is larger than
the source volume. The extra space on the target volume is simply not written to. However, if
an attempt is made to reverse the relationship (so that the source becomes the target), this
attempt will fail because now the source is larger than the target. Clearly the best way to
avoid this is to ensure the source and target are exactly the same size.
When fixed block volumes are created on the DS8000, the user is given three size choices:
ds The number of bytes allocated will be the requested capacity value
times 230 .
ess The number of bytes allocated will be the requested capacity value
times 109 .
blocks The number of bytes allocated will be the requested capacity value
times 512 bytes (since each block is 512 bytes).
When creating volumes that are going to be either RMC source or RMC target between
DS6000 and DS8000, the recommended volume creation method is to use -type ds, as this
creates binary sized volumes that will not result in any wasted extent space. However, if the
planned partner device is an ESS 800, then you should only use -type ess or -type blocks.
Creating matching fixed block volumes

In our example we are going to create volumes 1405 to 1407 on the DS8000 in Extent Pool
P0 to use as RMC target volumes. We can use the following DS CLI command:
mkfbvol -extpool P0 -cap 2 -type ds 1405-1407
Example 33-3 shows the resulting volumes. Each volume shows in the cap (2^30B) column
as being 2 GB and 4194304 blocks.
Example 33-3 Creating binary sized volumes

dscli> mkfbvol -extpool P0 -cap 2 -type ds 1405-1407
Date/Time: 3 November 2005 19:47:28 IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-7503461
CMUC00025I mkfbvol: FB volume 1405 successfully created.
dscli> lsfbvol -lss 14
ID accstate datastate configstate deviceMTM datatype extpool cap (2^30B) cap (10^9B) cap (blocks)
====================================================================================================
1400 Online Normal Normal 2107-900 FB 512 P0 - 10.0 19531264

1405 Online Normal Normal 2107-900 FB 512 P0 2.0 - 4194304
dscli>
Checking the block count at older code levels

Prior to DS8000 code bundle 6.0.500.31 or DS6000 code bundle 6.0.600.9, fixed block
volumes created using decimal sizes (as in their size is listed in the 10^9 column), could be
up to 32 k bytes larger than intended. If you are creating new volumes on a DS6000 or
DS8000 that are at or later than this code bundle, you do not need to worry about the block
count. However, if you created volumes below this code level, and you foresee the possibility
that you may want to use these volumes as targets in a DS8000 to DS6000 PPRC pair, then
you should check the block count to ensure the volume sizes truly match. To do this you need
to perform an lsfbvol on both the source and the target device. Then take note of the block
count of the source and target volumes. If the block counts do not match, then you must
either:
򐂰 Remove the incorrectly sized volume with rmfbvol and create it again (assuming you are
now at the higher bundle).
򐂰 Create a new volume for the purposes of RMC (again, assuming you are now at the
higher bundle).
򐂰 Use the larger volume only as a RMC target.
Using a spreadsheet to check expected versus actual block count

You can also use a formula to calculate the correct block size for any decimally sized volume.
You can use this in a spreadsheet to determine if your DS6000 or DS8000 has any volumes
that are slightly too large. The formula you would use is:
=INT((INT(size*10000000000/512)+63)/64)*64
Where the size parameter in the formula is the decimal (10^9B) volume size. You could place
the output of lsfbvol into a spreadsheet and calculate the expected block count for each
volume using the contents of the 10^9 (decimal) cap column. Then compare the column that
has the output of the formula to the block column to ensure the two match. In Table 33-1 the
output of the lsfbvol command has been modified and placed into a spreadsheet. The
formula has been placed into the Expected Block Size column. It uses the values in the
(10^9B) column as an input to determine the correct block size. One volume, 6002, is 64
blocks larger than it should be. If that volume is to be used for RMC, it should be deleted and
re-created (provided the data on it can be moved).
Table 33-1 Using a spreadsheet to calculate correct block size

ID Data extpool cap 2^30B (10^9B) (blocks) Expected
type block size
1000 FB 512 P2 - 5 9765632 9765632
1001 FB 512 P2 - 5 9765632 9765632
1002 FB 512 P2 - 5 9765632 9765632
1003 FB 512 P0 - 5 9765632 9765632
4400 FB 512 P0 - 10 19531264 19531264
5100 FB 512 P3 - 4.5 8789120 8789120
6000 FB 512 P2 - 9.4 18359424 18359424
6002 FB 512 P2 - 5.5 10742272 10742208
Note: Do not apply this formula to System i volumes or volumes that only have a size in the
2^30 column. This is because System i volumes will be correctly sized using 520 byte
blocks, while volumes that are binary sized (their size is listed in the 2^30) will also have
the correct number of blocks.
33.2.8 Establishment errors on newly created volumes

When volumes are created on an ESS 800, DS6000, or DS8000, an internal process is used
to format the volumes. If you attempt to use these volumes as RMC or FlashCopy targets
while this process is occurring, the establishment will fail.
In Example 33-4, a fixed block volume ID 1808 was created on the DS6800. We then
immediately attempted to use it as a RMC target. This attempt fails with the message, as
shown. Do not be concerned about the FlashCopy comment; this is normal. Wait for the
volume initialization process to finish and try again. The initialization time will vary, depending
on the size of the volume.
Example 33-4 Establishing a RMC (PPRC) pair where the target is still being formatted
dscli> mkpprc -remotedev IBM.1750-1300247 -type mmir 1406:1808
ate/Time: 3 November 2005 23:02:31 IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-7503461
MUN03117E mkpprc: 1406:1808: Unable to establish FlashCopy or Remote Mirror and Copy pair.
A FlashCopy initialization is in progress.
33.3 RMC: Establishing paths between DS6000 and DS8000

To create an RMC relationship between a DS8000 and a DS6000 we first need to create
logical paths (over the physical Fibre Channel connections). If we are using the DS GUI, then,
provided we added the other machine’s Storage Complex, we can establish paths in either
direction (DS6000 to DS8000 or DS8000 to DS6000).
33.3.1 Decoding port IDs

When viewing port IDs in the DS GUI or DS CLI, the port IDs can be decoded to show which
physical port on the DS8000 or DS6000 is in use.
For DS8000, port IDs look like I0000 or I0123, which actually breaks out as IEECP. The EE is
the enclosure number (minus 1), C is the card slot number (minus 1), and P is the port
number on the card. So I0123 is enclosure 2 (1+1), slot 3 (2+1), port 3.
For DS6000, port IDs range from I0000 to I0001 and I0100 to I0103, which actually breaks
out as IEECP. The EE is the controller number (00 or 01), C is the card slot number (always
zero), and P is the port number on the card (0 to 3). So I0103 is controller 1, card slot 0,
port 3.

33.3.2 Path creation using the DS GUI
Path creation between a DS6000 and a DS8000 is no different than the process used to
create paths between two DS8000s or two DS6000s. See 12.5, “Metro Mirror paths and links”
on page 121.
Paths in the reverse direction

The example above showed how to create a path from the DS8000 to the DS6000. In many
cases it may be likely that you will need paths from DS6000 to the DS8000. Regardless,
having established paths in one direction you can then establish paths in the opposite
direction. Fibre Channel allows for bi-directional mirroring over the same physical path.
Adding or deleting paths

You can add additional paths to an LSS pair by simply creating more paths. To remove a path,
just display the Paths screen, select the relevant source machine and LSS, select the path
you wish to delete, and use the delete option from the Select Action pull-down.
33.3.3 Establish logical paths between DS8000 and DS6000 using DS CLI
You can also use the DS CLI to establish logical paths. One additional step is that you need
to know the WWNN of the target Storage Image. The three commands we will use to
establish a path are lssi, lsavailpprcport, and mkpprcpath.
Determining the remote device WWNN

First, we need to determine the WWNN of the remote storage device. If you started the DS
CLI by connecting to the DS8000 HMC, then the remote storage device is the DS6000. If you
started the DS CLI by connecting to the DS6000, then the remote Storage Image is the
DS8000.
How to determine the DS6000 WWNN using DS GUI

To do this:
3. Click Storage units.
4. Click in the Select column for the DS6000 Storage Unit, and from the Select Action
pull-down, choose Properties.
5. The DS6000 Storage Unit WWNN will be displayed on the subsequent properties screen.
How to determine DS6000 WWNN using DS CLI

In Example 33-5 we show how to display the WWNN of a Storage Image using the lssi
command.
Example 33-5 Determine the WWNN of a DS6000

dscli> lssi
============================================================================
- IBM.1750-1300247 IBM.1750-1300247 511 500507630EFFFE16 Online Enabled
How to determine the DS8000 WWNN using DS GUI
To do this:
3. Click Storage images (not the Storage Unit).
4. Click in the Select column for the DS8000 Storage Image, and from the Select Action
5. The DS8000 Storage Image WWNN will be displayed on the subsequent properties
screen.
How to determine DS8000 WWNN using DS CLI

In Example 33-6 we show how to display the WWNN of a DS8000 Storage Image using the
lssi command.

dscli> lssi IBM.2107-7503461
============================================================================
- IBM.2107-7503461 IBM.2107-7503460 932 5005076303FFC08F Online Enabled
Listing the available ports using the DS CLI

Having determined the remote devices WWNN, we can now display the ports that are
available to establish RMC (PPRC) paths. In Example 33-7 we are logged onto the DS8000
using the DS CLI, so the remote device is the DS6000. We display ports available to establish
paths between LSS 14 on the DS8000 and LSS 18 on the DS6000.
Example 33-7 Displaying available ports for PPRC path establishment

dscli> lsavailpprcport -remotedev IBM.1750-1300247 -remotewwnn 500507630EFFFE16 14:18
=============================
I0001 I0103 FCP
I0131 I0003 FCP
Establishing the logical paths using DS CLI

Having determined which ports are available, for each LSS pair that we wish to copy and
mirror between, we now establish a one-way path using the mkpprcpath command. We can
then display established paths using the lspprcpath command.
In Example 33-8, we first establish a single path between LSS 14 on the DS8000 and LSS 18
on the DS6000. We then display the paths available for LSS 14 on the DS8000.
Example 33-8 Using DS CLI to establish PPRC paths

dscli> mkpprcpath -remotedev IBM.1750-1300247 -remotewwnn 500507630EFFFE16 -srclss 14
-tgtlss 18 I0001:I0103
dscli> lspprcpath 14

=========================================================
14 18 Success FF18 I0001 I0103 500507630EFFFE16
In Example 33-9 we add an additional path between LSS 14 on the DS8000 and LSS 18 on
the DS6000. Take careful note though, we include the existing path when creating a new
path. Otherwise the existing path is removed and only the new path will be available for use.
Example 33-9 Establishing additional paths using DS CLI

dscli> mkpprcpath -remotedev IBM.1750-1300247 -remotewwnn 500507630EFFFE16 -srclss 14
-tgtlss 18 I0001:I0103 I0131:I10003
=========================================================
To establish paths where the DS6000 is the source, we should connect to the DS6000 using
the DS CLI and follow the same process, but specify the DS8000 as the remote device.
Important: When you connect using the DS CLI to the DS8000 HMC, the DS8000 is the
local device and the DS6000 is the remove device. If you connect using the DS CLI to the
DS6000, then the DS6000 is now the local device and the DS8000 is the remote device.
Attention: The rmpprcpath command will remove all paths between the source and target
LSSs. To just reduce the path count, use the mkpprcpath command specifying only the
paths you wish to continue using.
33.4 Managing Metro Mirror or Global Copy pairs

Having established paths, we can now establish volume pairs.
33.4.1 Managing Metro Mirror or Global Copy pairs using the DS GUI
Establishing and managing Metro Mirror and Global Copy pairs between a DS6000 and a
DS8000 using the DS GUI is no different than using the DS GUI to establish pairs between
two DS8000s or DS6000s. See 14.4, “DS Storage Manager GUI” on page 136, and 22.3, “DS
Storage Manager GUI” on page 291.
33.4.2 Managing Metro Mirror pairs using the DS CLI

Having established paths, we can now establish volume pairs. In this example we show how
you can establish Metro Mirror volume pairs between a DS8000 and a DS6000 with the DS
CLI. In this example we create two Metro Mirror pairs. Volumes 1401 and 1402 from the
DS8000 are the source volumes, and the target volumes on the DS6000 are 1805 and 1806.
In Example 33-10 we create two pairs using the mkpprc command. We then list them using
the lspprc command. We then remove one pair using the rmpprc command.
Example 33-10 Creating Metro Mirror pairs using the DS CLI

dscli> mkpprc -remotedev IBM.1750-1300247 -type mmir -mode full 1405:1805 1406:1806
created.
created.
dscli> lspprc 1405-1406
===========================================================================================
dscli> rmpprc -remotedev IBM.1750-1300247 1405:1805
relationship 1405:1805:? [y/n]
y
withdrawn.
In Example 33-10 we connected to the DS8000 HMC using the DS CLI, so the DS6000 is the
remote device. If we wish to establish pairs where the DS6000 is the source device, we need
to connect to the DS6000 using the DS CLI. This will make the DS8000 the remote device.
Important: If you specify the wrong -remotedev or you are using a profile where a different
remote device to the one you intended to work on is specified, you may get an error
message, CMUN03057, saying you are specifying an invalid subsystem ID. This may be
because you are specifying the wrong remote device serial number. If you have multiple
possible remote devices, do not place a remotedev entry in your DS CLI profile.
33.4.3 Managing Global Copy pairs using DS CLI

you can establish Global Copy volume pairs between a DS8000 and a DS6000 with the DS
CLI. In this example we create two Global Copy pairs. Volumes 1405 and 1406 from the
DS8000 are the source volumes, and the target volumes on the DS6000 are 1805 and 1806.
the lspprc command. We then pause one pair using the pausepprc command.
Example 33-11 Using DS CLI to manage Global Copy

dscli> mkpprc -remotedev IBM.1750-1300247 -type gcp 1405:1805 1406:1806
created.
created.
===========================================================================================

dscli> pausepprc -remotedev IBM.1750-1300247 1405:1805
dscli> lspprc 1405
===========================================================================================
dscli>
33.5 Managing DS6000 to DS8000 Global Mirror

The establishment of Global Mirror between a DS8000 and a DS6000 is achieved using the
same methods as explained in Chapter 24, “Global Mirror examples” on page 305.
33.5.1 Managing Global Mirror pairs using the DS CLI

In Example 33-12 we establish a Global Copy pair between volume 1805 on the DS6000 and
volume 1405 on the DS8000. We then create session 30 for LSS 14. We then create a Global
Mirror using session 30 and LSS 14.
Example 33-12 Establishing Global Mirror using DS CLI

dscli> lspprc 1405
dscli> mkpprc -remotedev IBM.1750-1300247 -type gcp -tgtread -mode full 1405:1805
dscli> lspprc 1405
==================================================================================================
dscli> mkremoteflash -dev IBM.1750-1300247 -conduit IBM.2107-7503461/14 -record -nocp 1805:1806
dscli> mksession -lss 14 -volume 1405 30
dscli> lssession 14
===========================================================================================================
14 30 Normal 1405 Join Pending Primary Copy Pending Secondary Simplex True
dscli> mkgmir -lss 14 -session 30
dscli> showgmir 14
ID IBM.2107-7503461/14
Master Count 1
Copy State Running
Current Time 11/03/2005 21:09:32 EST
CG Time -
Subordinate Count 0
33.6 Managing DS6000 and DS8000 FlashCopy

The establishment of a FlashCopy pair on a DS6000 using the DS8000 DS GUI is no different
than establishing a DS8000 FlashCopy pair; see 8.5, “FlashCopy management using the DS
SM” on page 80. The establishment of a FlashCopy pair on a DS8000 using the DS6000 DS
GUI is again no different than establishing a DS6000 FlashCopy pair. For the DS CLI you
connect directly to the relevant DS SMC or DS HMC so it is also not different.
33.6.1 Creating a remote FlashCopy on a DS6000 using the DS CLI

It is also possible to use the DS CLI to create a remote FlashCopy on a DS6000 where the
source RMC device is a DS8000 or vice versa. In Example 33-13, we connect using the DS
CLI to a DS8000. We determine there are established paths from LSS 14 on the DS8000 to
LSS 18 on the DS6000. We create a Metro Mirror pair from volume 1405 on the DS8000 to
volume 1805 on the DS6000. We then create a remote FlashCopy on the DS6000 between
DS6000 volumes 1805 and 1806. We then remove the remote FlashCopy.
Example 33-13 Creating a remote FlashCopy where the DS6000 is the remote target
=========================================================
dscli> mkremoteflash -dev IBM.1750-1300247 -conduit IBM.2107-7503461/14 -persist -nocp 1805:1806
dscli> lsremoteflash -dev IBM.1750-1300247 -conduit IBM.2107-7503461/14 1805:1806
ID SrcLSS SequenceNum ActiveCopy Recording Persistent Revertible SourceWriteEnabled
===========================================================================================================
1805:1806 18 0 Disabled Disabled Enabled Disabled Enabled
dscli> rmremoteflash -dev IBM.1750-1300247 -conduit IBM.2107-7503461/14 1805:1806
CMUC00179I rmremoteflash: Are you sure you want to remove the remote FlashCopy pair {0}? [y/n]:y
CMUC00180I rmremoteflash: Removal of the remote FlashCopy volume pair 1805:1806 has been initiated
successfully. Use the lsremoteflash command to determine when the relationship is deleted.

You could also reverse the scenario and use the DS6000 as the source machine in a PPRC
pair and then create the remote FlashCopy on the DS8000.
Note: Commands that work with remote FlashCopies use the -dev parameter to define the
machine on which the FlashCopy is to be performed. Other commands, such as mkpprc
commands, will refer to this same device as the remote device, using -remotedev.
However, since a FlashCopy has to be sent to the remote site and then performed locally
on that site, the use of the -dev parameter to refer to the remote machine is correct.
34
Chapter 34. Interoperability between DS8000

and ESS 800
In this chapter we show the interoperability between the Copy Services functions in the
DS8000 and the ESS 800. This chapter contains the following sections:
򐂰 DS8000 and ESS 800 Copy Services interoperability
򐂰 Preparing the environment
򐂰 RMC: Establishing paths between DS8000 and ESS 800
򐂰 Managing Metro Mirror or Global Copy pairs
򐂰 Managing ESS 800 Global Mirror
򐂰 Managing ESS 800 FlashCopy

34.1 DS8000 and ESS 800 Copy Services interoperability
Copy Services operations are supported between the DS8000 and the IBM Enterprise
Storage Server Model 800 (ESS 800) and Model 750. For the rest of this chapter, all
references to the ESS 800 also apply to the ESS 750.
Note: On the ESS 800, Remote Mirror and Copy (RMC) is called Peer-to-Peer Remote
Copy (PPRC). All references to PPRC are interchangeable with RMC.
34.2 Preparing the environment

Before starting Copy Services operations in a mixed DS8000/ESS 800 environment, you
need to ensure this environment is set up correctly.
34.2.1 Minimum microcode levels

To manage the ESS 800 Copy Services from the DS8000, you need to have your IBM
Service Representative install licensed internal code Version 2.4.3.65 or later on the ESS
800, and DS8000 code bundle 6.0.500.52 or later on the DS8000.
34.2.2 Hardware and licensing requirements

To establish PPRC pairs between an ESS 800 and a DS8000, the ESS 800 must have a
PPRC license and the DS8000 must have a Remote Mirror Copy (RMC) license.
The ESS 800 must also have Fibre Channel adapters that have connectivity to the DS8000.
ESCON adapters cannot be used to mirror an ESS 800 to a DS8000. You cannot have a
Copy Services relationship between a 2105 E20 or F20 and a DS8000. You could, however,
have a cascaded installation where an ESS F20 was mirrored to an ESS 800 that was then
mirrored to a DS8000. The ESS F20 to ESS 800 relationship would have to be managed with
ESS management tools such as the ESS CLI or the ESS Web Copy Services GUI.
34.2.3 Network connectivity

Network connectivity requirements depend on how you want to manage the environment.
For FlashCopy management:

򐂰 If you want to use the DS8000 DS GUI to manage FlashCopy on the ESS 800, then you
need network connectivity between the DS8000 HMCs and the ESS 800 Copy Services
servers.
򐂰 If you want to be able to use the DS CLI to manage FlashCopy on the ESS 800 then you
need network connectivity between the machine on which you are running the DS CLI and
at least one of the ESS 800 Copy Services servers.
For creating Remote Mirror and Copy (RMC), also known as Peer-to-Peer Remote Copy
(PPRC), paths and pairs:
򐂰 If you wish to use the ESS 800 as a PPRC source machine, then you will need network
connectivity to the ESS 800 Copy Services servers, either from a machine running the DS
CLI, or from the DS8000 HMC.
򐂰 If, however, the ESS 800 will purely be a remote target for PPRC, and you do not plan to
ever use it as a source machine and you do not intend to use the DS GUI to manage the
pairs and paths, then you do not need to have network connectivity to the ESS 800 Copy
Services servers. This is because all path and pair establishment is done by connecting to
the source machine (which would be the DS8000). This setup is not recommended
because it is less flexible.
34.2.4 Create matching user IDs and passwords

When you want to use the DS CLI or DS GUI to perform Copy Services operations, you need
to authenticate with a valid user ID and password. When you use the DS GUI to perform an
operation that will require it to issue a command to an ESS 800, it needs to authenticate with
the ESS 800. To do this it uses the DS user ID and password that you used to log on to the
GUI with. This means that this user ID and password needs to be defined in the ESS
Specialist. This task needs to be manually performed. If instead of the DS GUI, you only use
the DS CLI, then you will be logging onto the ESS 800 Copy Services server directly, so the
requirement will not be there. For simplified management you may still want to create a
matching user ID and password.
Create a user ID on the DS8000

The first step is to log on to the DS8000 HMC using the DS GUI and create a user ID that is in
either the admin, op_storage or op_copy_services groups. Log off and then log on with that
user ID and change the initial password.
Create a user ID on the ESS 800

Once you have created a DS user ID, you need to create a matching user ID using the ESS
800 Web Specialist:
1. Start a Web Browser and connect to the IP address of either ESS 800 cluster.
2. Click ESS Specialist.
3. Log on with a ESS Specialist user ID that has admin privileges.
4. Click the Users tab.
5. Click Modify Users. Enter the user ID name and password you created in the DS CLI or
GUI. It must be given the Administration access level.
6. Click Add to move the user ID to the right hand box.
7. Click Perform Configuration Update and wait for the completion message.
Important: The ESS Web Copy Services user IDs are not used by the DS CLI or DS GUI.
You only need to create a matching ESS Specialist user ID.
34.2.5 Updating the DS CLI profile

If you plan to use the DS CLI to manage your ESS 800, you can create a profile to simplify
connection, commands, and scripted operations. Add extra lines as shown in Example 34-1.
Example 34-1 Possible modification to a DS CLI profile

# ESS 800
# hmc1 is the Copy Services server A
hmc1: 10.0.0.100
# devid is the serial number of the ESS 800 - note there are only 5 digits after the 2105
devid: IBM.2105-22399
# remotedevid is the serial number of the DS8000
remotedevid: IBM.2107-7503461
# Username is a user created on the ESS Specialist that matches the userid on the DS8000
Chapter 34. Interoperability between DS8000 and ESS 800 513

username:admin
# The password for the admin user id. Placing it here is not very secure.
password:passw0rd
# The password file is created using the managepwfile and is a better way to manage this.
# pwfile:security.dat
Putting the password in the profile is not very secure (because it is stored in plain text), but it
can be more convenient. A password file can be created using the managepwfile command
and is a better way to manage this. After creating the password file (which by default is called
security.dat), you can remove the password from the profile and instead specify the pwfile
file.
The command to create a passwd file in this example is:

managepwfile -action add -name admin -pw passw0rd
A simple method when you have multiple machines to manage is to create multiple profile
files. Then when starting the DS CLI, you can specify the profile file you wish to use with the
-cfg parameter. In a Windows environment you could have multiple Windows batch files (BAT
files), one for each machine you wish to manage. The profile shown in Example 34-1 on
page 513 could be saved in the C:\program files\ibm\dscli\profile directory as
2105source.profile. Then you create a simple Windows BAT file with three lines, as shown in
Example 34-2.
Example 34-2 Windows BAT file to start a specific profile

title DS CLI Local 2105 22399 Remote IBM.2107-7503461
cd C:\Program Files\ibm\dscli\profile
dscli -cfg 2105source.profile
We save the BAT file onto the Windows desktop and start it by double-clicking the icon. The
DS CLI will open and will start the specified profile. By creating a BAT file and profile for each
machine, we can simplify the process of starting the DS CLI. We can also specify the profile
when starting the DS CLI from inside a script, or when running the DS CLI in script mode.
34.2.6 Adding the Copy Services Domain

If you wish to use the DS GUI to manage FlashCopy on an ESS 800, or Remote Mirror and
Copy paths and pairs between an ESS 800 and a DS8000, you have to add the ESS Copy
Services domain to the DS HMC. You need the IP address of the ESS 800 Copy Services
servers (server A and, if desired, server B). You can get these by connecting with a Web
Browser to either cluster of the ESS, and clicking the Copy Services tab. They will be
displayed on the first window you see.
You add an ESS Copy Services domain with the DS Storage Manager to the DS8000 as
follows:
4. Select Add 2105 Copy Services Domain.
Figure 34-1 shows the Add 2105 Copy Services Domain panel.
Figure 34-1 Add 2105 Copy Services Domain
5. Enter the IP address of the ESS Copy Services server A in the Server 1 IP address box. If
you have a Server B, select the Define a second Copy Services server box and enter
the Server B IP address in the Server 2 IP address box. However, if Server B is running on
a 2105-F20, you should not define it.
Having added the ESS 800 Copy Services domain to the DS GUI, you are now able to use
the DS GUI to create paths and Remote Mirror and Copy pairs, where the ESS 800 is the
source device. You can also use the DS GUI to manage FlashCopy pairs on the ESS 800.
Note: The steps to add the ESS 800 Copy Services Domain to a DS8000 cannot be
performed using the DS CLI. However, if you do not plan to use the GUI, this will not be an
issue.
Restriction: You cannot use the ESS Copy Services Server Web GUI to manage Copy
Services relationships between an ESS 800 and a DS8000. The Volumes tab will only
show ESS 800 volumes, not DS8000 volumes. The Paths tab will not shows paths between
an ESS 800 and a DS8000. It will only show paths between ESS 800s. If you are using
PPRC between a DS8000 and an ESS 800, all management of that PPRC relationship
must be done with the DS CLI or via the DS GUI.
Storage management
You cannot use the DS8000 DS GUI to perform storage configuration on an ESS 800.
Likewise, you cannot use the ESS 800 Web Specialist to perform storage configuration on a
DS8000. You can only perform Copy Services management tasks on the alternative device. If
you are logged onto the DS8000 DS GUI and wish to configure some storage on the ESS
800, you will need to log on to the ESS 800 Web Specialist.
34.2.7 Volume size considerations for RMC (PPRC)

When volumes are created on an ESS 800 they are sized using decimal gigabytes. This
means that when a request is made to a create a 10 GB volume, the ESS 800 allocates a
minimum of 10,000,000,000 bytes. This is a very important consideration when using PPRC
between a DS8000 and an ESS 800. In PPRC it is not an issue if the target volume is larger
than the source volume. The extra space on the target volume is simply not written to.
However, if an attempt is made to reverse the relationship (so that the source becomes the
target), this attempt will fail because now the source is larger than the target. Clearly the best
way to avoid this is to ensure the source and target are exactly the same size.

When fixed block volumes are created on the DS8000, the user is given three size choices:
ds The number of bytes allocated will be the requested capacity value
times 230 .
ess The number of bytes allocated will be the requested capacity value
times 109 .
blocks The number of bytes allocated will be the requested capacity value
times 512 bytes (since each block is 512 bytes).
The correct method is to determine the ESS volume size and then create fixed block volumes
that are sized using the -type ess parameter.
Determining ESS volume size

To view the ESS volume sizes you can use the ESS Specialist.
2. Click ESS Specialist.
3. Log on with a ESS Specialist user ID.
4. Click the Storage Allocation tab.
5. Click the Open Systems Storage tab.
6. Click Modify Volumes Assignments.
7. Take note of the value in the size column for the volumes you are interested in, as shown
in Figure 34-2.
Figure 34-2 Viewing ESS 800 volume size using ESS Specialist GUI
You can also use the ESS CLI to view the volume size, as shown in Example 34-3.
Example 34-3 View ESS 800 volume size using ESS CLI
C:\Program Files\ibm\ESScli>esscli -s 10.0.0.1 -u storwatch -p specialist list volume
Wed Nov 02 01:13:10 EST 2005 IBM ESSCLI 2.4.0
Volume Cap Units VolType LSS VS Serial Label

------ ----- ----- ------- --- ---- -------- -----
1000 1.1 GB FB 10 vs0 00022399 ***
1001 1.1 GB FB 10 vs0 00122399 ***
1002 1.1 GB FB 10 vs0 00222399 ***
1003 1.1 GB FB 10 vs0 00322399 ***
Creating matching fixed block volumes on the DS8000
In Figure 34-2 on page 516 and Example 34-3 on page 516 we listed four volumes, which are
all shown as being 1.1 GB. In our example we are going to create volumes 1400 to 1403 on
the DS8000 in Extent Pool P0 to use as RMC (PPRC) target volumes. We can use the
following DS CLI command:
mkfbvol -extpool P0 -cap 1.1 -type ess 1400-1403
Example 34-4 shows the resulting volumes. Each volume shows in the cap (10^9B) column
as being 1.1 GB.
Example 34-4 Resulting volumes

dscli> mkfbvol -extpool p0 -cap 1.1 -type ess 1400-1403
Date/Time: 27 October 2005 22:24:15 IBM DSCLI Version: 5.1.0.204 DS: IBM.2107-7503461
dscli> lsfbvol
Name ID accstate datastate configstate deviceMTM datatype extpool cap (2^30B) cap (10^9B) cap (blocks)
===========================================================================================================
Checking the block count at older code levels

Prior to DS8000 code bundle 6.0.500.31, fixed block volumes created on the DS8000 could
be up to 32 k bytes larger than the equivalent sized volume on the ESS 800. If you are
creating new volumes on a DS8000 that are at or above this code bundle, you do not need to
worry about the block count. However, if you created volumes below this code level, and you
foresee the possibility that you may want to use these DS8000 volumes as targets in a
DS8000 to ESS 800 PPRC pair, then you should check the block count to ensure the volume
sizes truly match. To do this you need to first check the block count of the volume on the ESS
800. To do this you need to open the ESS Copy Services server:
2. Click Copy Services.
3. Log on with a ESS Specialist user ID that has admin privileges.
4. Click the Volumes tab.
5. In the source pull-down, choose the LSS for which your particular volume is located.
6. When the volumes in that LSS are displayed, single left-click a particular volume to
highlight it.
7. Now click the Information tab on the bottom right-hand corner of the window.
8. In the subsequent window, take note of the sectors count. This is actually the block count.
In Figure 34-3 on page 518 the example is 2148480 sectors (or blocks).

9. Now take note of the block count from the output of the DS CLI command lsfbvol (for the
proposed source volume on the DS8000). In Example 34-4 on page 517, the block count
is 2148480, which is an exact match.
Figure 34-3 Displaying block count using Copy Services server
10.If the block counts do not match, then you must either:
– Remove the volume with rmfbvol and create it again (assuming you are now at bundle
6.0.500.31 or later).
– Create a new volume for the purposes of PPRC (again, assuming you are now at
bundle 6.0.500.31 or later).
– Use this volume only as a PPRC target, since the equivalent ESS 800 source volume
is slightly smaller.
Using a spreadsheet to check expected versus actual block count

You can also use a formula to calculate the correct block size for any DS8000 volume that has
to be matched in size to an ESS 800 volume. You can use this in a spreadsheet to determine
if your DS8000 has any volumes that are slightly too large. The formula you would use is:
=INT((INT(size*10^9/512)+63)/64)*64
Where the size parameter in the formula is the ESS 800 volume size. You could place the
output of lsfbvol into a spreadsheet and calculate the expected block size for each volume
using the contents of the 10^9 (decimal) cap column. Then compare the column that has the
output of the formula to the block column, to ensure the two match. In Table 34-1 the output of
the lsfbvol command has been modified and placed into a spreadsheet. The formula has
been placed into the Expected Block Size column. It uses the values in the (10^9B) column as
an input to determine the correct block size. One volume, 6002, is 64 blocks larger than it
should be. If that volume is to be used for PPRC, it should be deleted and re-created
(provided the data on it can be moved).
Table 34-1 Using a spreadsheet to calculate correct block size

ID Data extpool cap 2^30B (10^9B) Blocks Expected
type block size
1000 FB 512 P2 - 5 9765632 9765632
1001 FB 512 P2 - 5 9765632 9765632
1002 FB 512 P2 - 5 9765632 9765632
1003 FB 512 P0 - 5 9765632 9765632
4400 FB 512 P0 - 10 19531264 19531264
5100 FB 512 P3 - 4.5 8789120 8789120
6000 FB 512 P2 - 9.4 18359424 18359424
6002 FB 512 P2 - 5.5 10742272 10742208
Note: Do not apply this formula to System i volumes or volumes that only have a size in the
2^30 column. This is because System i volumes will be correctly sized using 520 byte
blocks, while volumes that are binary sized (their size is listed in the 2^30) will also have
the correct number of blocks.
One option when creating fixed block volumes is to use the parameter -type blocks. The
number of blocks could be calculated using the methods shown in Figure 34-3 on page 518
or Table 34-1 on page 518. An example command would be:
mkfbvol -extpool P0 -type blocks -cap 10742208 1404
34.2.8 Volume address considerations on the ESS 800

On the ESS 800, open systems volume IDs are given in an 8-digit format, xxx-sssss, where
xxx is the LUN ID and sssss is the serial number of the ESS 800. An example of this is shown
in Figure 34-2 on page 516, where the volumes shown are 000-22399 to 003-22399. These
volumes are open systems, or fixed block volumes. When referring to them in the DS CLI, you
must add 1000 to the volume ID, so volume 000-22399 is volume ID 1000 and volume
001-22399 is volume ID 1001. This is very important because on the ESS 800, the following
address ranges are actually used:
0000 to 0FFF CKD volumes (4096 possible addresses)
1000 to 1FFF Open systems fixed block LUNs (4096 possible addresses)
If we intend to use FlashCopy to copy ESS LUN 000-22399 onto 001-22399 using the DS
CLI, we must specify 1000 and 1001. If instead we specify 0000 and 0001, we will actually
run the FlashCopy against CKD volumes. This may result in an unplanned outage on the
System z environment that was using CKD volume 0001.
34.2.9 Establishment errors on newly created volumes

When volumes are created on an ESS 800, DS6000, or DS8000, an internal process is used
to format the volumes. If you attempt to use these volumes as PPRC or FlashCopy targets
while this process is occurring, the establishment will fail. Wait for the volume initialization
process to finish and try again. The initialization time will vary depending on the size of the
volume.
34.3 RMC: Establishing paths between DS8000 and ESS 800

To create a PPRC relationship between a DS8000 and an ESS 800 we first need to create
logical paths (over the physical Fibre Channel connections). If we are using the DS GUI, then
provided we added the ESS Copy Services Domain to the DS8000, we can establish paths in
either direction (ESS 800 to DS8000 or DS8000 to ESS 800). If the ESS Copy Services
domain was not added, then paths can only be established where the DS8000 is the source.

34.3.1 Decoding port IDs
When viewing port IDs in the DS GUI or DS CLI, the port IDs can be decoded to show which
physical port on the DS8000 or ESS 800 is in use. For DS6000 or DS8000, port IDs look like
I0000 or I0123, which actually breaks out as IEECP. The EE is the enclosure number (minus
1), C is the card slot number (minus 1), and P is the port number on the card. So I0123 is
enclosure 2 (1+1), slot 3 (2+1), port 3.
The ESS 800 port IDs do not follow the same rule. To decode them, take the last two digits in
the port ID and then use Figure 34-4. So port ID I0020 is the adapter in host bay 2, slot 1 and
port ID I00AC is the adapter in host bay 4, slot 4.
00 04 08 0C 20 24 28 2C 80 84 88 8C A0 A4 A8 AC
Slot 1 Slot 2 Slot 3 Slot 4 Slot 1 Slot 2 Slot 3 Slot 4 Slot 1 Slot 2 Slot 3 Slot 4 Slot 1 Slot 2 Slot 3 Slot 4
Host Bay 1 Host Bay 2 Host Bay 3 Host Bay 4
Figure 34-4 ESS 800 Port IDs decoded
34.3.2 Path creation using the DS GUI

In this example we show how to establish two RMC (PPRC) paths with the DS Storage
Manager from DS8000 LSS 14 to ESS LSS 10.
Tip: Open systems LSSs on an ESS 800 are always LSS 10 to LSS 1F. If you select ESS
800 LSS 00 to ESS 800 LSS 0F, you are working with System z LSSs on the ESS 800.

3. Click Paths.
4. Select the Storage Complex, Storage Unit, and Storage Image from which you want to
create PPRC paths. For this path, this machine will be providing the source LSS.
5. Click Create; see Figure 34-5.
Figure 34-5 Path panel
6. You will now be prompted to select the source LSS of the DS8000 from which you want to
establish the PPRC paths. You then click Next. In this example we want to establish
PPRC paths from LSS 14. See Figure 34-6.
Figure 34-6 Select source LSS
7. Now you have to select the target LSS. First select in the Storage Complex pull-down
menu the Storage Complex for the ESS. Then select from the Storage Unit pull-down
menu the appropriate Storage Unit and the LSS from the Storage Unit. When you are
finished, click Next to continue. In Figure 34-7 the target LSS on the ESS is LSS 10.
Figure 34-7 Select target LSS

8. The next panel shows you which Fibre Channel ports are available to establish Metro
Mirror and Copy paths; see Figure 34-8. Select the ports and click Next. Because you
want to establish two paths from the DS8000 to the ESS you must select both I/O ports
from the DS8000 that are available for RMC. You will only see physical connections that
actually exist. To get connections listed, the HBAs in the DS8000 and ESS 800 have to
either be zoned together or directly connected.
Figure 34-8 Select source I/O ports
9. In the next panel (see Figure 34-9) you have to select for each source I/O port from the
DS8000 a target I/O port on the ESS. When done click Next.
Figure 34-9 Select target I/O ports
10.On the next two panels you will be asked whether you want built a Consistency Group and
to verify the information you entered during the process. After you verify the information
you entered, click Finish to establish the RMC paths.
11.Figure 34-10 shows the two RMC paths we have established for LSS 14. To manage the
established paths (for example, delete path) you can select the path you want to manage
and then select from the pull-down menu the appropriate action you want to perform.
Figure 34-10 Path panel
Paths in the reverse direction

The previous example showed how to create a path from the DS6000 to the ESS 800. In
many cases it may be likely that you will need paths from the ESS 800 to the DS8000.
Regardless, by having established paths in one direction, you can then establish paths in the
opposite direction. Fibre Channel allows for bi-directional mirroring over the same physical
path.
Adding or deleting paths

You can add additional paths to an LSS pair by simply creating more paths. To remove a
path, just display the Paths screen, select the relevant source machine and LSS, select the
path you wish to delete, and use the delete option from the Select Action pull-down.
34.3.3 Establish logical paths between DS8000 and ESS 800 using DS CLI
You can also use the DS CLI to establish logical paths. One additional step is that you need
to know the WWNN of the target storage image. The three commands we will use to establish
a path are lssi, lsavailpprcport, and mkpprcpath.
Determining the remote device WWNN

First, we need to determine the WWNN of the remote storage device. If you started the DS
CLI by connecting to the DS8000 HMC, then the remote storage device is the ESS 800. If you
started the DS CLI by connecting to the ESS 800, then the remote Storage Image is the
DS8000.
Determining the DS8000 WWNN using DS GUI

To do this:
3. Click Storage images (not the Storage Unit).
4. Click in the Select column for the DS8000 Storage Image, and from the Select Action

5. The DS8000 Storage Image WWNN will be displayed on the subsequent properties
screen.
Determining DS8000 WWNN using DS CLI

In Example 34-5 we show how to display the WWNN of a Storage Image using the lssi
command.

dscli> lssi IBM.2107-7503461
============================================================================
- IBM.2107-7503461 IBM.2107-7503460 932 5005076303FFC08F Online Enabled
Determining the WWNN of the ESS 800 using the ESS Specialist
To do this:
1. Point your browser at the IP address of either cluster of the ESS 800.
2. On the opening screen, click the ESS Specialist button.
3. You will receive a logon prompt. Log on to the ESS Specialist using a ESS Specialist user
ID and password.
4. On the welcome screen, you will see the WWNN, as shown in Figure 34-11. You need to
write it down.
Figure 34-11 Using ESS 800 Specialist GUI to display the ESS 800 WWNN
Determining the WWNN of the ESS 800 using the ESS CLI
If you have the ESS CLI installed on a PC then you can use the list server command to
display the ESS 800 WWNN.
Note: This is not the DS CLI. ESS CLI is a separate software package that you can get
from your IBM Service Representative if you do not already have it.
An example of the command syntax is shown in Example 34-6. This technique has the
advantage that you can copy and paste the output.
Example 34-6 Using ESS CLI to display the ESS 800 WWNN
C:\Program Files\ibm\ESScli>esscli -u storwatch -p specialist -s 10.0.0.1 list server
Tue Nov 01 03:28:59 EST 2005 IBM ESSCLI 2.4.0
Server Model Mfg WWN CodeEC Cache NVS Racks
---------- ----- --- ---------------- -------- ----- ----- -----
2105.22399 800 013 5005076300C09517 2.4.3.79 32GB 2GB 1
Listing the available ports using the DS CLI

Having determined the remote devices WWNN, we can now display the ports that are
available to establish PPRC paths. In Example 34-7 we are logged onto the DS8000 using
the DS CLI, so the remote device is the ESS 800.
Example 34-7 Displaying available ports for PPRC path establishment

dscli> lsavailpprcport -remotedev IBM.2105-22399 -remotewwnn 5005076300C09517 00:00
=============================
I0000 I000C FCP
I0000 I00AC FCP
I0002 I0088 FCP
I0003 I0084 FCP
I0130 I000C FCP
I0130 I00AC FCP
I0241 I00A4 FCP
I0311 I0024 FCP
Note: When issuing commands that refer to an ESS 800, the ESS 800 serial number is
only five digits, not seven as the DS6000 or DS8000 are. So in our examples, the serial
number syntax we use is IBM.2105-22399, not IBM.2105-1322399.
Establishing the logical paths using the DS CLI

Having determined which ports are available for each LSS pair that you wish to copy and
mirror between, you now establish a one-way path with the mkpprcpath command. You can
then display established paths using the lspprcpath command.
In Example 34-8, we first establish a single path between LSS 12 on the DS8000 and LSS 15
on the ESS 800. We then display the paths available for LSS 12 on the DS8000.
Example 34-8 Using the DS CLI to establish RMC (PPRC) paths

dscli> mkpprcpath -remotedev IBM.2105-22399 -remotewwnn 5005076300C09517 -srclss 12
-tgtlss 15 I0000:I000C
=========================================================
12 15 Success FF21 I0000 I000C 5005076300C09517
In Example 34-9 we add an additional path between LSS 12 on the DS8000 and LSS 15 on
the ESS 800. Note that we include the existing path when creating a new path. Otherwise, the
existing path is removed and only the new path will be available for use.
Example 34-9 Establishing additional paths using the DS CLI

dscli> mkpprcpath -remotedev IBM.2105-22399 -remotewwnn 5005076300C09517 -srclss 12
-tgtlss 15 I0000:I000C I0130:I000C

=========================================================
12 15 Success FF21 I0000 I000C 5005076300C09517
12 15 Success FF21 I0130 I000C 5005076300C09517
To establish paths where the ESS 800 is the source, we should connect to the ESS 800 using
the DS CLI and follow the same process, but specify the DS8000 as the remote device.
Important: When you connect using the DS CLI to the DS8000 HMC, the DS8000 is the
local device and the ESS 800 is the remove device. If you connect using the DS CLI to the
ESS 800, then ESS 800 is now the local device and the DS8000 is the remote device.
Attention: The rmpprcpath command will remove all paths between the source and target
LSSs. To just reduce the path count, use the mkpprcpath command specifying only the
paths you wish to continue using.
34.4 Managing Metro Mirror or Global Copy pairs

Having established paths, we can now establish volume pairs.
34.4.1 Managing Metro Mirror or Global Copy pairs using the DS GUI
In this example we show how you can establish Metro Mirror volume pairs between a DS8000
and an ESS 800 with the DS Storage Manager. In this example we create two Metro Mirror
pairs. Volumes 1401 and 1402 from the DS8000 are the source volumes, and the target
volumes are 1000 and 1001 from the ESS. We would follow the same method to set up a
Global Copy pair, except that Global Copy would be selected in Figure 34-17 on page 529.
2. Click Copy services.
4. Select the Storage Complex, Storage Unit, and Storage Image on which the Metro Mirror
source volumes are.
5. Click Create; see Figure 34-12.
Figure 34-12 Metro Mirror
The next panels guide you through the process of creating Metro Mirror volume pairs. In
the last panel, you have the opportunity to review the information you entered before you
establish the Metro Mirror pairs. Also, during the process, you can, at any time, go back to
modify specifications that you have already done, or you can cancel the process.
6. You have the option to choose between Automated volume assignment and Manual
volume pair assignment; see Figure 34-13. If you click the Automated volume pair
assignment, the first selected source volume is paired automatically with the first selected
target volume. If you click Manual volume pair assignment, you must select each
specific target volume for each selected source volume. In this example we assign the
volume pairs manually.
Figure 34-13 Pairing method

7. The source volumes we want to use are on the DS8000 in LSS 14. Select both volumes
and click Next; see Figure 34-14. Note that you also have the option from this panel to
create paths.
Figure 34-14 Select source volume
8. In the next panel, select the storage complex and storage unit for the LSS with the target
volumes. Then select the target volume for the first source volume. When you are
finished, click Next, as shown in Figure 34-15. To expand your choices, you must select
the small blue boxes.
Figure 34-15 First Metro Mirror target volume
9. Next select the target volume for the second source volume, as shown in Figure 34-16. To
expand your choices, you must select the small blue boxes.
Figure 34-16 Second Metro Mirror target volume
10.In the next panel you can specify various copy options, as shown in Figure 34-17. In this
example we selected Metro Mirror under Define relationship type and Perform initial
copy. If you wish to instead use Global Copy, this is where you select it.
Figure 34-17 Copy options

11.The Verify panel opens. Verify the information that you entered, and if everything is
correct, click Finish to establish the Metro Mirror volume pairs. Figure 34-18 shows the
Metro Mirror pairs for LSS 06 that we established. To manage the established pairs (for
example, suspend pair) you can select the volume pair that you want to manage and then
select the appropriate action that you want to perform from the menu.
Figure 34-18 Managing Metro Mirror pairs
34.4.2 Managing Metro Mirror pairs using the DS CLI

you can establish Metro Mirror volume pairs between a DS8000 and an ESS 800 with the DS
CLI. In this example we create two Metro Mirror pairs. Volumes 1401 and 1402 from the
DS8000 are the source volumes, and the target volumes on the ESS are 1000 and 1001.
the lspprc command. We then remove one pair using the rmpprc command.
Example 34-10 Creating Metro Mirror pairs using DS CLI

dscli> mkpprc -remotedev IBM.2105-22399 -type mmir -mode full 1401:1000 1402:1001
created.
created.
===========================================================================================
dscli> rmpprc -remotedev IBM.2105-22399 1401:1000
relationship 1401:1000:? [y/n]
:y
withdrawn.
In Example 34-10 we connected to the DS8000 HMC using the DS CLI, so the ESS 800 is the
remote device. If we wish to establish pairs where the ESS 800 is the source device, we need
to connect to the ESS 800 using the DS CLI. This will make the DS8000 the remote device.
Important: If you specify the wrong -remotedev or you are using a profile where a different
remote device from the one you intended to work on is specified, you may get an error
message, CMUN03057, saying you are specifying an invalid subsystem ID. This may be
because you are specifying the wrong remote device serial number. If you have multiple
potential remote devices, do not specify a remotedev in your DS CLI profile.
34.4.3 Managing Global Copy pairs using the DS CLI

you can establish Global Copy volume pairs between a DS8000 and an ESS 800 with the DS
CLI. In this example we create two Global Copy pairs. Volumes 1401 and 1402 from the
DS8000 are the target volumes, and the source volumes on the ESS are 1000 and 1001.
In Example 34-10 on page 530 we create two pairs using the mkpprc command. We then list
them using the lspprc command. We then pause one pair using the pausepprc command.
Example 34-11 Using DS CLI to manage Global Copy

dscli> mkpprc -remotedev IBM.2107-7503461 -type gcp 1000:1401 1001:1402
created.
created.
===========================================================================================
dscli> pausepprc 1000:1401
dscli> lspprc 1000
===========================================================================================
34.5 Managing ESS 800 Global Mirror

The establishment of Global Mirror between a DS8000 and an ESS 800 is achieved using the
same methods as explained in Chapter 24, “Global Mirror examples” on page 305.

34.5.1 Managing Global Mirror pairs using the DS CLI
In Example 34-12 we establish a Global Copy pair between volume 1000 on the ESS 800 and
volume 1401 on the DS8000 and a remote FlashCopy to support it. We then create session
12 for LSS 10. We then create a Global Mirror using session 12 and LSS 10.
Example 34-12 Establishing Global Copy using DS CLI

dscli> lspprc 1000
dscli> mkpprc -type gcp -tgtread -mode full 1000:1401
dscli> lspprc 1000
==================================================================================================
dscli> mkremoteflash -dev IBM.2107-7503461 -conduit IBM.2105-22399/10 -record -nocp 1402:1403
dscli> mksession -lss 10 -volume 1000 12
dscli> lssession 10
===========================================================================================================
10 12 Normal 1000 Join Pending Primary Copy Pending Secondary Simplex True
dscli> mkgmir -lss 10 -session 12
dscli> showgmir 10
ID IBM.2105-22399/10
Master Count 1
Copy State Running
Current Time 11/02/2005 21:40:47 EST
CG Time 11/02/2005 21:40:47 EST
Subordinate Count 0
34.6 Managing ESS 800 FlashCopy

If there is a FlashCopy license for the ESS 800, it is possible to manage ESS FlashCopy
using the DS CLI or SM GUI. The establishment of a FlashCopy pair on an ESS 800, using
the DS GUI, is no different than establishing a DS8000 FlashCopy pair; see 8.5, “FlashCopy
management using the DS SM” on page 80. The establishment of a FlashCopy pair on an
ESS 800 using the DS CLI is also no different; see 8.3, “Local FlashCopy using the DS CLI”
on page 66. In a situation where the ESS 800 is the target device in a PPRC relationship, it is
also possible to create remote FlashCopies on the ESS 800 using the mkremoteflash
command. In each case, creating, changing, and removing the FlashCopy is done using the
same commands.
34.6.1 Creating an ESS 800 FlashCopy using the DS GUI

The steps to create an ESS 800 FlashCopy using the DS GUI are:
3. Click FlashCopy.
4. From the Storage Complex menu, choose the ESS 800 Copy Services Domain.
5. From the Storage Units menu, select the ESS that you wish to perform the FlashCopy on.
6. Make selections from the Resource type and Specify LSS pull-downs.
7. From the Select Action menu, select Create, as shown in Figure 34-19.
Figure 34-19 Creating ESS 800 FlashCopy using DS GUI
8. Choose between A single source with a single target and A single source with
multiple targets and click Next.

9. Select the source volumes, as shown in Figure 34-20. Note that some of the columns have
Not Applicable for 2105 in them. This is normal.
Figure 34-20 Selecting ESS 800 source volumes for FlashCopy
10.Select the target volumes and click Next.

11.Select what options you plan to use and click Next.
12.When you get the verification screen, review your selections, and click Finish.
34.6.2 Creating an ESS 800 FlashCopy using DS CLI

Start the DS CLI and connect to the ESS 800. Then issue DS CLI commands as per normal.
An example is shown in Example 34-13.
Example 34-13 Using DS CLI to create an ESS 800 FlashCopy

dscli> mkflash -nocp -record -persist 1001:1002
dscli> lsflash 1001
ID SrcLSS SequenceNum Timeout ActiveCopy Recording Persistent Revertible SourceWrite TargetWrite
===========================================================================================================
1001:1002 10 0 45 Disabled Enabled Enabled Disabled Enabled Enabled
dscli> rmflash 1001:1002
CMUC00144W rmflash: Are you sure you want to remove the FlashCopy pair 1001:1002:? [y/n]:y
34.6.3 Creating a remote FlashCopy on an ESS 800 using DS CLI
It is also possible to use DS CLI to create a remote FlashCopy on an ESS 800 where the
source PPRC device is a DS8000. In Example 34-14, we connect using the DS CLI to a
DS8000. We determine there are established paths from LSS 14 on the DS8000 to LSS 10
on the ESS 800. We create a Metro Mirror pair from volume 1401 on the DS8000 to volume
1001 on the ESS 800. We then create a remote FlashCopy on the ESS 800 between ESS
800 volumes 1001 and 1002. We then remove the remote FlashCopy.
Example 34-14 Creating a remote FlashCopy where the ESS 800 is the remote target
=========================================================
14 10 Success FF16 I0000 I000C 5005076300C09517
14 10 Success FF16 I0130 I000C 5005076300C09517
dscli> mkremoteflash -dev IBM.2105-22399 -conduit IBM.2107-7503461/14 -persist -nocp 1001:1002
dscli> lsremoteflash -dev IBM.2105-22399 -conduit IBM.2107-7503461/14 1001:1002
ID SrcLSS SequenceNum ActiveCopy Recording Persistent Revertible SourceWrite TargetWrite
===========================================================================================================
1001:1002 10 0 Disabled Disabled Enabled Disabled Enabled Enabled
dscli> rmremoteflash -dev IBM.2105-22399 -conduit IBM.2107-7503461/14 1001:1002
CMUC00179I rmremoteflash: Are you sure you want to remove the remote FlashCopy pair {0}? [y/n]:y
CMUC00180I rmremoteflash: Removal of the remote FlashCopy volume pair 1001:1002 has been initiated
successfully. Use the lsremoteflash command to determine when the relationship is deleted.
You could also reverse the scenario and use the ESS 800 as the source machine in a PPRC
pair and then create the remote FlashCopy on the DS8000.
Note: Commands that work with remote FlashCopies use the -dev parameter to define the
machine on which the FlashCopy is to be performed. Other commands, such as mkpprc
commands, refer to this device as the remote device, using -remotedev. However,
because a FlashCopy must be sent to the remote site and then performed locally there, the
use of the -dev parameter to refer to the remote machine is correct.

Part 9
Part 9 Solutions
In this part we discuss solutions offered by IBM to assist you in the management, automation,
and control of your Copy Services implementation on the DS8000. We provide an overview of
the solutions, discuss the options available, and give some examples of using the solutions.

35
Chapter 35. IBM TotalStorage Productivity

Center for Replication
The IBM TotalStorage Productivity Center for Replication is an automated solution to provide
a management frontend to Copy Services. This chapter describes the IBM strategic solution
to manage disaster recovery solutions based on copy service functions in DS6000 and
DS8000 storage servers. For details refer to the IBM Redbooks and product manuals
mentioned at the end of this introduction paragraph.
In addition to the DS6000 and DS8000, the IBM TotalStorage Productivity Center for
Replication or TPC for Replication can also help manage copy services for the SAN Volume
Controller (SVC), as well as the ESS 800 when using FCP links for PPRC paths between
ESS 800s or between ESS 800 and DS6000 and DS8000. This also applies to FlashCopy.
For installation and configuration details refer to Powering SOA with IBM Data Servers,
SG24-7259.
You must refer to the following manuals when implementing and managing TPC for
Replication configurations:
򐂰 IBM TotalStorage Productivity Center for Replication User’s Guide, SC32-0103
򐂰 IBM TotalStorage Productivity Center for Replication Installation and Configuration Guide,
SC32-0102
򐂰 IBM TotalStorage Productivity Center for Replication Command-Line Interface User’s
Guide, SC32-0104
Note: The IBM TotalStorage Productivity Center for Replication is completely independent
from TPC for Disk, Data, and Fabric.

35.1 IBM TotalStorage Productivity Center
The IBM TotalStorage Productivity Center (TPC) is a suite of software products. It is designed
to support customers in monitoring and managing their storage environments.
Design and development emphasis for TPC is on scalability and standards. The approach
based on open standards allows TPC to manage any equipment or solution implementation
that follow the same open standards.
Figure 35-1 shows a split set of products with three components based on former Tivoli
products and a package with two components designed and developed to facilitate storage
replication solutions.
IBM TotalStorage Productivity Center (TPC)

Standard Edtion Replication
Replication
TPC for Disk TPC for Data TPC for Fabric Replication
Two Site BC
BC Business Continuity
Figure 35-1 TPC software products suite
All three Tivoli products provide a single source and single management tool to cover the
tasks of the storage manager or network manager in their daily business.
򐂰 TPC for Disk is software based on open standard interfaces to query, gather, and collect
all available data necessary for performance management.
򐂰 TPC for Data focuses on data management and addresses aspects related to information
life-cycle management.
򐂰 TPC for Fabric is a management tool to monitor and manage a SAN fabric.
TPC for Replication, as a member of the TPC product family, is a standalone package and
does not build on anything related to the TPC Standard Edition.
It comes in two flavors:

򐂰 TPC for Replication, which includes support for:
– FlashCopy
– Planned failover and restart (one direction) for MM and GM
򐂰 TPC for Replication Two Site Business Continuity (BC), which includes support for:
– FlashCopy
– Planned and unplanned failover and failback for MM and GM
– High availability (2 TPC Replication servers)
In the following paragraphs we cover the basic structures and features of TPC for Replication
and TPC for Replication Two Site BC. This is to explain and discuss Copy Services topics
with the DS6000 and DS8000.
35.2 Where we are coming from
Since the advent of copy services functions with IBM storage servers a framework is required
to handle and manage disk storage environments and the various combinations of software,
firmware, and hardware used for replication.
To ensure and guarantee data consistency at the remote or backup site it is even more
important to provide a framework around copy services functions that helps achieve that data
consistency and ease of management.
Other aspects are automation and scalability. Early solution attempts turned out to have
weaknesses that would often surface during actual implementation in real production
environments. Tools offered on an as is basis are not suitable for managing production
environments. Other solutions are perfectly suited for certain host server platforms but cannot
manage other server platforms.
This led to the design and implementation of a framework that is platform independent and
provides the required production attributes such as scalability, stability, and security, and
provides as a standard product also offering the guarantee to be serviced and maintained.
Furthermore, the concept has the potential for enhancements and can evolve over time and
according to user requirements.
TPC for Replication builds on all previous experiences with storage management tools and
framework proposals to organize all aspects of copy services for disaster recovery solutions.
TPC for Replication also address the need for other solutions, which involve copy services
functions available with the DS6000 and DS8000 like data or volume migration, data center
movements, or other projects that require copying or moving data between similar devices
such as the DS6000, DS8000, and ESS 800.
35.3 What TPC for Replication provides

TPC for Replication is designed to help administrators manage copy services. This applies
not only to the copy services provided by DS6000 and DS8000 but also to copy services
provided by the ESS 800 and SAN Volume Controller (SVC).
In more details, the highlights of functions provided by TPC for replication:

򐂰 Manage Copy Services for the IBM System Storage DS6000 and DS8000.
򐂰 Extend support for ESS 800 to manage Copy Services.
򐂰 Provide disaster recovery support with failover/failback capability for the:
– ESS 800
– DS6000
– DS8000
򐂰 Monitor Copy Services performance.
򐂰 The optional high availability feature allows us to continue replication management even
when one TPC for Replication server goes down.
The basic functions of TPC for Replication provide management of:

򐂰 FlashCopy
򐂰 Metro Mirror
򐂰 Global Mirror
Chapter 35. IBM TotalStorage Productivity Center for Replication 541

Note that TPC for Replication does not support Global Copy.
Figure 35-2 is a screen capture from the TPC for Replication GUI showing the different
session types, or Copy Services functions, supported.
Figure 35-2 Management of Copy Services Functions supported by TPC for Replication
TPC for Replication is designed to simplify management of Copy Services by:

򐂰 Automating administration and configuration of Copy Services functions with wizard-based
session and copy set definitions.
򐂰 Provide simple operational control of Copy Services tasks, which includes starting,
suspending, and resuming Copy Services Tasks.
򐂰 Offer tools to monitor and manage Copy Services sessions.
Note that TPC for Replication also manages FlashCopy and Metro Mirror for IBM SAN
Volume Controller (SVC).
35.4 Copy Services terminology

Although Copy Services terminology is discussed in a previous part of this book, we have
included here, for convenience and to keep it in context with TPC for Replication, a brief
review.
35.4.1 FlashCopy
FlashCopy is a point-in-time copy of a source volume mapped to a target volume. It offers
various options including the choice between a COPY option, for a background copy through
physical I/O operations in the storage server back-end, or a NOCOPY option.
FlashCopy is available on the DS6000 and DS8000 as well as on the ESS 800. FlashCopy is
also available on the DS4000 family and with the SAN Volume Controller. Note, however, that
the DS400 and SVC use different implementations of the FlashCopy function that are not
compatible with the DS6000. DS8000, and ESS 800 FlashCopy.
35.4.2 Metro Mirror
Metro Mirror (MM) is a synchronous data replication mechanism and was previously called
Peer-to-Peer Remote copy (PPRC). I/O completion is signaled when the data is secured on
both the local storage server and the remote storage server. Metro Mirror is popular for two
site disaster recovery solutions.
Metro mirror is available in any combination between DS6000, DS8000, and ESS800.
Synchronous data replication is also available on the DS4000 family and with the SVC, but
again not directly compatible with Metro Mirror on the DS6000, DS8000, and ESS 800.
35.4.3 Global Copy

Global Copy (GC) is an asynchronous data replication mechanism and used to be called
PPRC for eXtended Distance (PPRC-XD). Asynchronous in this context means that the data
transmission between local and remote storage servers is separated from signaling the I/O
completion. Note that GC does not guarantee that the arriving writes at the local site are
applied to the remote site in the same sequence. Therefore, it does not provide data
consistency. GC is suited for data migration over any distance.
Global Copy is possible in any combination between DS6000, DS8000, and ESS 800.
Restriction: TPC for Replication does not provide an interface to manage a Global Copy
environment.
35.4.4 Global Mirror

Global Mirror (GM) is an asynchronous data replication over any distance. It is a combination
of GC and FlashCopy complemented by unique firmware features. The firmware drives the
copy operations in a fully autonomic fashion to guarantee consistent data at any time at the
remote site. This holds also for the requirement to provide consistent data at any time, which
can be spread over more than a single storage server.
GM is also the base for a two-site disaster recovery solution involving any DS6000, DS8000,
or ESS 800. The function as such is also possible between products of the DS4000 family
and the SVC.
35.4.5 Metro/Global Mirror

Metro/Global Mirror (MGM) is a three-site disaster recovery solution.
MGM is a combination of MM and GM. This is a synchronous data replication between two
sites in a metropolitan area and an asynchronous copy of the same data at a different remote
site, which can be located at any distance.
In a MGM configuration, a software-based solution is also needed to provide automation and

to guarantee that any pre-determined action happens as quickly as possible and always with
the same results when a trigger causes this action to happen. However, MGM configurations
and operations are currently not supported by TPC for Replication. A software utility, named
MGM Utility Toolkit for ICKDSF, and TSO can be used to manage MGM from a z/OS-based
environment. For details, refer to The IBM TotalStorage DS8000 Series: Copy Services with
IBM Eserver zSeries, SG24-6787.

Restrictions: Currently MGM is only supported in environments that include DS8000 and
ESS 800 storage servers. It is not supported with the DS6000.
MGM configurations and operations are currently not supported by TPC for Replication.
35.4.6 Failover/failback terminology

There is often a misconception about what failover/failback commands do. Figure 35-3
illustrates the failover/failback two-step process. The process is as follows:
1. When the replication process between primary (P) and secondary (S) is suspended as a
result of a planned or unplanned outage, you may want to access the S volumes
immediately without any checking on the P volumes.
The failover command always points to the S volumes and turns the S volumes from
SECONDARY state into a PRIMARY SUSPEND state, making the volumes immediately
available to the application. This state change for the S volumes at the remote site
happens without any communication or any checking of the P volumes at the local site.
The P volumes at the local site keep the status they had before the failover command was
issued.
Local DS6000 / DS8000 Remote DS6000 / DS8000
Suspends due to planned

or unplanned event
A A
A
Primary
Primary
Primary
Replication direction
A
Primary
Primary
Primary
P
Primary S
Primary
Primary Secondary
A A 1. Failover command
A
Primary
Primary A
Primary
Primary
Primary Primary
P
Primary
P
Primary
2. Failback command
PRIMARY SUSPENDED
A Resynchronisation
A
A
Primary
Primary A
Primary
Primary
Primary Primary
SECONDARY PENDING S
Primary
PRIMARY PENDING P
Primary
SECONDARY DUPLEX PRIMARY DUPLEX
OR 2. Failback command
A A
Resynchronisation
A
Primary
Primary A
Primary
Primary
Primary Primary
PRIMARY PENDING P
Primary
SECONDARY PENDING S
Primary
PRIMARY DUPLEX SECONDARY DUPLEX
Figure 35-3 Failover/failback terminology
2. Once the outage is over and you decide to resume replicating data between, you have a
choice as whether to re-synchronize from the remote site to the local site or vice versa.
This depends on what is required after updating one of the two sites or even updating both
sites after the suspend action.
a. When you decide to re-synchronize from the remote site to the local site the failback
command is pointing to the remote volumes. You need a PPRC path from the remote
site to the local site before you can perform the corresponding commands. Doing this
will resynchronize both sites with the level of data as it is at the remote site. Only the
data that changed at the remote site, from the moment the replication was suspended
and until the failover happened, is replicated from remote to local volumes.
Figure 35-3 on page 544 assumes a Metro Mirror (MM) environment that puts the
corresponding MM pair in PENDING state during re-synchronization, and once
everything is replicated the MM pair goes into DUPLEX state.
b. If you want to re-synchronize both sites with the data level at the local site, issue the
failback command towards the volumes at the local site. This replicates the changed
data since the suspend from the local site to the remote site. Also, this resets the
tracks that were potentially changed at the remote site after the failover.
Figure 35-3 on page 544 assumes a MM environment that places the MM pair in
PENDING state during the re-synchronization phase. Once all changed data is
replicated, the MM pair goes into DUPLEX state.
The option to re-synchronize from either site is possible due to the availability of change
bitmaps maintained by the DS6000 or DS8000 at both sites.
35.5 TPC for Replication terminology

TPC for Replication manages and integrates not only the DS6000 and DS8000 but also the
SAN Volume Controller. In search of a common terminology and to describe the functions for
the different disk storage servers in a common way, new terms, different from those normally
used in the context of Copy Services with the ESS 800, DS6000, and DS8000, are introduced
here.
35.5.1 TPC for Replication copy set

A copy set in TPC for Replication is a set of volumes that have a copy of the same data. In
PPRC terms this is, for example, a PPRC primary volume and a PPRC secondary volume.

Figure 35-4 shows three Metro Mirror Copy pairs. In TPC for Replication each pair is
considered as a copy set. A copy set here contains two volumes.
A
P3
Primary
Primary
PPRC path S3 Copy Set
Primary Secondary
PPR link
A
P2
Primary
Primary
PPRC path S2 Copy Set
Primary Secondary
PPR link
A
Primary
A
Primary
Primary
P1
Primary PPRC path S1
Primary Copy Set
Primary Secondary
Figure 35-4 TPC for Replication - Metro Mirror copy sets
Figure 35-5 represents two copy sets. In this illustration, each copy set contains three
volumes, which indicates a Global Mirror configuration. Note that the third volume is a kind of
a journal volume and is the FlashCopy target volume covering for data consistency in a
Global Mirror setup.
Global Copy
FlashCopy
A A
Primary Primary
P2
Primary S2
Primary
Primary Secondary
Copy Set
Primary
Primary
J2
Tertiary
PPRC links
Global Copy
A A FlashCopy
Primary
Primary
P1
Primary S1
Primary
Primary Secondary
Primary Copy Set
Primary
J1
Tertiary
Local site Remote Site

Figure 35-5 TPC for Replication - Global Mirror copy sets
35.5.2 TPC for Replication session

TPC for Replication uses a session concept, which is similar to what Global Mirror for
System z (XRC) uses.
A session is a logical concept that gathers multiple copy sets, representing a group of
volumes with the requirement to provide consistent data within the scope of all involved
volumes. Commands and processes performing against a session apply these actions to all
copy sets within the session. Again this is similar to a Global Mirror for System z (XRC)
session.

Figure 35-6 shows an example of two storage servers at the local site and two corresponding
storage servers at the remote site. The example further assumes that a Metro Mirror relation
is established to replicate data between both sites.
Session A
P3
Primary
Primary PPRC path S3 Copy Set
Primary Secondary
1
A
P2 PPRC path Copy Set
Primary
Primary S2
Prim ary Secondary
A A
Primary Copy Set
Primary
P1
Primary PPRC path Primary
S1
Primary
Primary Secondary
Session A
Primary
PB
Primary PPRC path
A
Primary
Primary
SB
Primary
Copy Set
Primary
2
Secondary
A A
Primary Copy Set
Primary
PA
Primary PPRC path Primary
SA
Primary
Primary Secondary
Local site Remote Site

Figure 35-6 TPC for Replication - session concept
Metro Mirror primary volume P1 in the second storage server, and volumes P2 and P3 in the
first storage server with their corresponding Metro Mirror secondary volumes are grouped
together in a session, session 1. Such a session can contain copy sets that span across
storage server boundaries.
Metro Mirror primary volumes PA and PB with their counterparts at the remote site belong to
a different session, session 2.
Note that all application-dependent copy sets must belong to the same session to guarantee
a successful management and to provide consistent data across all involved volumes within a
session. In this context we recommend having application-dependent volume granularity
within the scope of an LSS. Or, in other words, volumes or copy sets that require consistent
data and can be subject to a freeze trigger ought to be grouped in one or more LSSs and that
LSS must not contain other copy sets. This because the scope of the freeze function is at the
LSS level and affects all volumes within that LSS.
35.6 TPC for Replication session types
There are two TPC for Replication editions, which include different levels of Copy Services
functions.
35.6.1 TPC for Replication Basic Edition

The Basic Edition includes Metro Mirror and Global Mirror, in addition to FlashCopy.
However, Metro Mirror and Global Mirror are only configured to replicate data in a
uni-directional fashion.
FlashCopy
FlashCopy includes all functional flavors of FlashCopy. This includes FlashCopy with:
򐂰 Background copy or no background copy.
򐂰 Convert no background copy to background copy.
򐂰 Add persistent to a FlashCopy relationship.
򐂰 Establish incremental FlashCopy to apply only changed data since a previous FlashCopy
operation.
Metro Mirror
Metro Mirror is between a local site (site 1) and a remote site (site 2) in a uni-directional
fashion. It also allows switching the replication direction from site 2 to site 1. This may happen
through failover/failback operations.
Global Mirror
Global Mirror implies three volumes for each Global Mirror copy set. Again, this happens
between a local site (site 1) and a distant site (site 2) in a uni-directional way. It is possible to
reverse the replication direction but still only in a uni-directional fashion.
35.6.2 TPC for Replication Advanced Edition

The Advanced Edition has, in addition to the Basic Edition, the capability to replicate data in
either direction.
Metro Mirror
Metro Mirror may be established from site 1 to site 2 or vice versa. It allows also
failover/failback operations for any copy set at any time.
Global Mirror
The Advanced Edition allows the Global Mirror to operate in either direction at the same time
(bi-directional). It may also swap sites with failover/failback operations.
Global Mirror builds on three volumes per copy set. TPC for Replication allows also managing
a configuration that only replicates data through Global Copy in the opposite direction than
Global Mirror did before.

35.7 TPC for Replication session states
Again a session contains a group of copy sets (that is, RMC (PPRC) volume pairs or
FlashCopy pairs) that belong to a certain application. You may also consider it as a collection
of volumes that belong to a certain application or system with the requirement for
consistency.
Such a session may be in one of the following states:

򐂰 Defined
A session is defined and may already contain copy sets or still have no copy sets
assigned yet. However, a defined session is not yet started.
򐂰 Preparing
The session started already and is in the process of initializing, which may be, for
example, in the case of a first full initial copy for a Metro Mirror. It may also just re-initialize,
which, for example, may be the case for a resynchronization of a previously suspended
Metro Mirror. Once the initialization is complete, the session state changes to prepared.
򐂰 Prepared
All volumes within the concerned session completed the initialization process.
򐂰 Suspending
This is a transition state caused by either a suspend command or any other suspend
trigger that may be an error in the storage subsystem or loss of connectivity between sites.
Eventually the process to suspend copy sets ends and copying has stopped, which is
indicated by a session state of suspended.
򐂰 Suspended
Replicating data from site 1 to site 2 has stopped. Application writes to concerned
volumes in site 1 can continue (FREEZE and GO). An additional recoverable flag
indicates whether the data is consistent and recoverable.
򐂰 Recovering
A session is about to recover.
򐂰 TargetAvailable
The recover command has completed and the target volumes are write enabled and
available for application I/Os. An additional recoverable flag indicates whether the data is
consistent and recoverable.
Important: Do not manage through another software, Copy Service volume pairs which
are already managed through TPC for Replication.
35.8 Volumes in a copy set

With TPC for Replication, the role of a volume within Copy Services has been renamed to be
more generic and also to include other storage subsystems like SVC.
A copy set contains the volumes that are part of a Copy Services relationship. Metro Mirror
knows two volumes per copy set. Global Mirror requires three volumes per copy set.
FlashCopy again consists of two volumes per copy set.
Terminology is slightly different from what you may be used to. For example, Metro Mirror
uses a primary or source volume at the sending site and a secondary or target volume at the
receiving end. Such a pair is now called a copy set.
FlashCopy usually used to mention source and target volume to identify a FlashCopy pair, in
contrast to RMC, which designates volumes in that pair as primary and secondary volumes.
Again such a FlashCopy volume pair is now a copy set.
Global Mirror involves three volumes per copy set. Global Mirror relies on Global Copy and
FlashCopy. Therefore we have a Global Copy primary volume and a Global Copy secondary
volume. The Global Copy secondary volume has another role as FlashCopy source volume.
The third volume is then the FlashCopy target volume. All three volumes are part of a Global
Mirror copy set.
35.8.1 Host volume

A host volume is identical to what is called a primary or source volume in Copy Services. The
host designation represents the volume functional role from an application point of view. It is
usually connected to a host or server and receives read, write, and update application I/Os.
When a host volume becomes the target volume of a Copy Services function it is usually no
longer accessible from the application host. FlashCopy target volumes can be considered as
an exception.
35.8.2 Target volume

A target volume is what was also usually designated as a secondary volume. It receives data
from a host volume or another intermediate volume. It may also be an intermediate volume
like in a Global Mirror copy set.
35.8.3 Journal volume

A journal volume is currently the FlashCopy target volume in a Global Mirror copy set. It is
called a journal volume because it functions like a journal and holds the required data to
reconstruct consistent data at the Global Mirror remote site. When a session needs to be
recovered at the remote site, the journal volume is used to restore data to the last consistency
point.

35.9 TPC for Replication and scalability
From an architecture point of view there is no limit to the number of Copy Sets that may
belong to a single session. The implementation approach of managing copy sets and
sessions provides the scalability that very large installations require.
Local DS6000 / DS8000 Intermediate DS6000 / DS8000 Remote DS6000 / DS8000
A A A
APrimary
Primary APrimary
Primary APrimary
Primary
A Primary
Primary A Primary
Primary A Primary
Copy Set
Primary A
Primary
Primary
Primary
Primary Session 1 Primary
Primary APrimary
Primary
APrimary
Primary
Primary
Primary
A A
A Primary
Primary A
A Primary
Primary Session 2 Primary A Primary
Primary
Primary Primary Primary
Primary Primary
A A
Primary Primary
Primary Primary
A A
A Primary
Primary Session 3 A Primary
Primary
Primary Primary
APrimary APrimary
Primary Primary
Primary Primary
A A A
Primary
Primary Session 4 Primary
Primary
Primary
Primary
Figure 35-7 TPC for Replication - four sessions
There is also no limit to the number of sessions that TPC for Replication can manage.
Figure 35-7 shows four sessions managing different copy sets. The first session, session 1,
implies a Metro/Global Mirror configuration that is not supported (at the time of writing this
book) by TPC for Replication.
The second session, session 2, is a Global Mirror session between an intermediate site and a
remote site with two copy sets.
The third session, session 3, is a Metro Mirror configuration between the local and
intermediate site.
The session at the bottom in Figure 35-7, session 4, is a Global Mirror session between the
local and remote sites.
35.10 TPC for Replication system and connectivity overview

TPC for Replication is an outboard approach and its software runs on a dedicated server, or
on two servers, for a high availability configuration.
Figure 35-8 shows how the TPC for Replication server connects to the storage server that is
going to be managed by TPC for Replication servers. It does not show the connectivity of the
TPC for Replication server to the network through which the user connects with the browsers
to the server itself.
Primary
Primary DB2 Websphere
Windows
Database Replication Manager Server Linux
AIX
IP Communication
Services
IP network
Ethernet ports
Series p Series p PowerPC PowerPC
FCP ports
A A
APrimary
Primary
APrimary
Primary
A Primary
Copy Set
Primary A A
Primary
Primary A
Primary
Primary
Copy Set
Primary A
Primary
A
Primary
Primary
Primary A
Primary
Primary
Primary
Primary
APrimary
Primary
Primary Primary
Primary
PPR FCP links
A A
A Primary A A
Primary A Primary A Primary
Primary
Primary
Primary
Primary
Primary Primary A Primary
Primary
Primary Primary Primary
Primary
DS8000 DS6000
Figure 35-8 TPC for Replication system overview
TPC for Replication server contains, besides the actual functional code, a DB2 UDB and
WebSphere® software.
IP communication services also contains an event listening capability to react to respective

trap messages from the storage servers. This provides a handle to plan for particular events
and provide pre-established scripts that are going to be triggered by a corresponding trap
message. This also includes a capability to distinguish between the different storage servers
that can be managed by the Replication Manager such as DS6000, DS8000, ESS 800, and
SVC. This approach has the potential to be enhanced as storage servers change over time
without touching the actual functional code and the involved database.

Figure 35-9 shows how the Replication Manager server connects to the DS6000 and
DS8000.
Replication Manager Server SMC
IP network
Ethernet ports
PowerPC PowerPC
server0 server1
server0 server1
System p System p DS6000
DS8000
Figure 35-9 Replication Manager server connectivity to DS6000 and DS8000
The actual connectivity between the TPC for Replication server and the storage servers is
based on Ethernet networks and connects to particular Ethernet ports in the System p™ in
the DS8000. This particular Ethernet card is a new card and slides into the first slot out of
these four slots in the p570. Note that Figure 35-9 shows the DS8000 rear view because
these slots are only accessible from the rear site of the DS8000.
New DS8000 Ethernet card feature codes

This new Ethernet card is required for TPC for Replication and is available for the following
DS8000 models:
򐂰 921, 922, 931, and 932 with feature code 1801 for the Ethernet adapter pair. Note that you
always need a pair of cards because one Ethernet card installs in sever0 and a second
card installs in server1.
򐂰 9A2 and 9B2 with feature code 1802 for the Ethernet adapter pair for the first LPAR.
򐂰 9A2 and 9B2 with feature code 1803 for the Ethernet adapter pair for the second LPAR.
These features are chargeable and carry a minimum monthly maintenance charge.
DS8000 Ethernet card installation and configuration considerations

This new Ethernet card may come already installed by manufacturing or may be installed
on-site.
To configure the Ethernet ports, Release 2 microcode is required for the concerned DS8000.
This is a code bundle that starts with 6.2.xxx.xx.
Port numbers on the first card are I9801 and I9802. This is the card that installs in server0.
Port numbers on the second card are I9B01 and I9B02. This is the card that installs in
server1. Note that only the first port on each card is currently used.
Communication through these ports uses static IP addresses. DHCP is not supported.
You may configure these new ports either through the GUI or the DSCLI.
The GUI provides a new panel to configure the required IP addresses. This panel is in the
GUI path of the storage image. Select a storage image, then select Configure network
ports from the Select Action pull-down (note that this option is only presented when the
network card for TPC for Replication is actually installed). This is shown in Figure 35-10. The
Configure Network ports panel will then display.
Figure 35-10 Select Configure network ports
The DSCLI provides the following commands to manage these new network ports:
򐂰 lsnetworkport -l
This command shows the server associations, physical port location, and all IP address
settings for all ports on the queried Storage Image Facility. See Example 35-2 on
page 556.
򐂰 shownetworkport
This shows the server association, physical location, and IP addresses for a particular port
on the Ethernet card. See Example 35-3 on page 556.

򐂰 setnetworkport
This command configures the network ports. Example 35-1 displays a command example
and configures the first port on the Ethernet card in server0.
Example 35-1 setnetoworkport command example

dscli> setnetworkport -dev IBM.2107-7520781 -ipaddr 9.155.86.128 -gateway
9.155.86.1 -subnet 255.255.255.0
-primary 9.64.163.21 -secondary 9.64.162.21 N9801
Example 35-2 shows an output example of lsnetworkport -l, which provides an overview of
all available Ethernet ports on the concerned storage image facility.
Example 35-2 Output of lsnetworkport command

dscli> lsnetworkport -l
Date/Time: 28 August 2006 13:19:55 CEST IBM DSCLI Version: 5.2.200.308 DS:
IBM.2107-7503461
ID IP Address Subnet Mask Gateway Primary DNS Secondary DNS State Server
Speed Type Location
==================================================================================
==================================================
I9801 9.155.50.53 255.255.255.0 0.0.0.0 9.64.163.21 9.64.162.21 Online 00
1 Gb/sec Ethernet-Copper U7879.001.DQD04X5-P1-C1-T1
I9802 0.0.0.0 0.0.0.0 0.0.0.0 9.64.163.21 9.64.162.21 Offline 00
1 Gb/sec Ethernet-Copper U7879.001.DQD04X5-P1-C1-T2
I9B01 9.155.50.54 255.255.255.0 0.0.0.0 9.64.163.21 9.64.162.21 Online 01
1 Gb/sec Ethernet-Copper U7879.001.DQD04WH-P1-C1-T1
I9B02 0.0.0.0 0.0.0.0 0.0.0.0 9.64.163.21 9.64.162.21 Offline 01
1 Gb/sec Ethernet-Copper U7879.001.DQD04WH-P1-C1-T2
Example 35-3 shows an output example of shownetworkport, which provides an overview of

all settings for a particular Ethernet port.
Example 35-3 Output of shownetworkport for a particular Ethernet port

dscli> shownetworkport i9801
Date/Time: 28 August 2006 13:20:00 CEST IBM DSCLI Version: 5.2.200.308 DS:
IBM.2107-7503461
ID I9801
IP Address 9.155.50.53
Subnet Mask 255.255.255.0
Gateway 0.0.0.0
Primary DNS 9.64.163.21
Secondary DNS 9.64.162.21
State Online
Server 00
Speed 1 Gb/sec
Type Ethernet-Copper
Location U7879.001.DQD04X5-P1-C1-T1
TPC for Replication server connectivity to the DS6000

For the DS6000 the Replication Manager server connects to the network that connects to the
PowerPC® servers in the DS6000. For the DS6000 this is the same network where the
Storage Management console connects to because the DS6000 controller or server card
contains only a single Ethernet port.
For the DS6000 there is only a single Ethernet port per cluster or server card. This port is
shared between the Replication Manager server or servers as well as with one or two Storage
Management consoles.
TPC for Replication configuration options

Please note that Figure 35-9 on page 554 outlines the basic connectivity idea. For high
availability you may consider a second Replication Manager server that connects to the same
IP network as the first server. Currently, the Replication Manager server connects only to one
Ethernet port out of the two ports on the Ethernet card, which must reside in the first slot of
the System p server in the DS8000.
Note that the internal and potential external HMCs connect to the DS8000 through different
ports than the Replication Manager servers.
The communication between the Replication Manager Server and the DS infrastructure is
direct, as shown in Figure 35-9 on page 554. It is interesting to note that this is different from
when the Replication Manager communicates with the SAN Volume Controller (SVC).
Between the Replication Manager Server and the SVC nodes is a SVC CIMON based
console, which is part of the standard SVC master console.
For more details refer to Powering SOA with IBM Data Servers, SG24-7259.

35.11 TPC for Replication monitoring and freeze capability
TPC for Replication does always use the consistency group attribute when you define PPRC
paths for Metro Mirror between a primary and a secondary storage server. This provides TPC
for Replication with the capability to freeze a Metro Mirror configuration when an incident
happens to guarantee consistent data at the secondary or backup site.
Replication Manager Server
FREEZE 3
2
LSS LSS
LSS
1 LSS
LSS LSS
LSS LSS
LSS LSS
LSS LSS
Primaries
Session Secondaries
Figure 35-11 TPC for Replication server freeze
TPC for Replication server listens to incidents from the storage servers and takes action
when being notified of a replication error from the concerned storage server.
Figure 35-11 implies a replication error in an LSS that belongs to the session. The TPC
server receives a corresponding SNMP trap message from the concerned storage server.
TPC for Replication server then issues a freeze command to all LSSs that are part of the
concerned session. This implies a suspend of all PPRC pairs or copy sets that belong to this
session. During this freeze process, write I/Os are held until the freeze process ends and the
TPC server communicates to the storage server to continue processing write I/O requests to
the concerned primary volumes in the session.
After that, write I/O can continue to the suspended primary volumes. However, both sites are
not in sync any longer but the data on the secondary site is consistent (power drop
consistent). This is a freeze-and-go policy.
35.12 TPC for Replication heartbeat
Because the connectivity between the TPC for Replication server and the storage servers
that the TPC server is managing can fail, the firmware in the storage server wait for a
heartbeat signal from the TPC server. TPC for Replication can enable this heartbeat in the
corresponding LSS for Metro Mirror sessions.

Ethernet ports
1 FCP ports
2 FREEZE
LSS LSS
LSS PPRC FCP links LSS
LSS LSS
Primaries
Session Secondaries
Figure 35-12 LSS heartbeat triggers freeze when connectivity to server fails
Figure 35-12 implies a failing connectivity between the RM server and a primary storage
server. Once this heartbeat is set and the storage subsystem cannot communicate its
heartbeat information to the RM sever, the storage server internally triggers a freeze to its
involved LSS and their primary volumes. Because this heartbeat is a timer scheduled
function, it is based on the Consistency Group time-out value of each LSS that contains
volumes that belong to the concerned session. This session-based heartbeat timer expires
after the lowest time-out value of all concerned LSSs.
With more than a storage server involved, the RM server issues freeze commands to all other
LSS pairs in the affected session once the heartbeat expires and the RM server could not
receive heartbeat information from any one of the involved storage servers.
The disconnected storage server or LSS resumes I/O after their Consistency Group time-out
has expired. LSS or storage servers that may still be connected to the RM server receive a
corresponding freeze command from the RM server followed by a run command when a
freeze-and-go policy has been selected.
Note that Extended Long Busy (ELB), automation window, and freeze period are synonyms
for Consistency Group time-out in the above context.

35.13 Supported platforms
TPC for Replication server can currently run under the following operating systems:
򐂰 Windows 2003 Server Edition with SP1
򐂰 Windows 2003 Enterprise Edition SP1
Figure 35-13 Windows 2003 Software level
򐂰 SUSE Linux Enterprise Server 9 SP2 (Note that SUSE Linux does not support the Two
Site BC configuration.)
򐂰 Red Hat Enterprise Linux RHEL4 AS 2.1 (For Replication Two Site BC: Red Hat
Enterprise Linux RHEL4 Update 1 SLES9 SP2)
򐂰 AIX 5.3 ML3
Note that for a TPC for Replication Two Site BC configuration that involves two servers, it is
possible to run TPC for Replication under two different operating system platforms.
For the latest software requirements refer to:

http://www.ibm.com/servers/storage/support/software/tpcrep/installing.html
35.14 Hardware requirements for TPC for Replication servers

For Windows and Linux we suggest using the following minimum hardware configuration:
򐂰 1.5 GHz Intel® (TM) Pentium® (R) III processor
򐂰 2 GB RAM memory
򐂰 10 GB free disk space
When TPC for Replication runs on AIX the following minimum hardware configuration is
suggested:
򐂰 Server p, IBM POWER4™ or IBM POWER5™ processor, 1 GHz
򐂰 2 GB RAM memory
򐂰 10 GB free disk space
Disk space is required to hold for data in DB2 databases and WebSphere Express
Application Server code besides the actual TPC for Replication server code.
You may refer to the following Web site for the latest information about hardware
requirements:
http://www-03.ibm.com/servers/storage/support/software/tpcrep/installing.html
35.15 TPC for Replication GUI

This section describes the graphical user interface (GUI) that is used to communicate with the
the replication server. Additional details can be found in the documents mentioned at the very
beginning of this chapter. See Chapter 35, “IBM TotalStorage Productivity Center for
Replication” on page 539
TPC for Replication provides a graphical user interface to manage and to monitor any Copy
Services configuration and Copy Services operations. This GUI is Web browser based and
does not rely on any other product like TotalStorage Productivity Center and IBM Director.
URL of GUI (RM server)
Figure 35-14 GUI URL determined during Replication Manager install time
The GUI is simply called through an Internet browser like MS Internet Explorer® or Mozilla
Firefox, is intuitive, easy-to-use and performs very well. The panel structure is not too deep
and allows you to quickly transition to any application through a hyperlink-based menu.

Figure 35-15 displays the TPC for Replication GUI and its components used between the
client on the user machine and the server component on the server machine.
Login
User machine Global Mirror settings

Configure sessions
Session States Browser / GUI Advanced tools
LAN
Data- Monitor Server

base
Hardware Lavers
IP network
Series p Series p PowerPC PowerPC
DS8000 DS6000
Figure 35-15 TPC for Replication GUI
The Web-based client that runs on the user’s machine contains the GUI client, a presentation
component. This software piece on the user’s machine is highly optimized to minimize data
traffic over the LAN to the RM server.
The GUI server component runs on the RM server machine that contains the interface to the
function code in the connected storage servers and is usually closely installed to the storage
servers that interface with the RM server.
35.15.1 Connect to the TPC for Replication GUI
Connect to the GUI by specifying the IP address of the TPC for Replication server in your
Web browser. This will present you with the sign-on panel, as shown in Figure 35-16. Once
you sign out from the RM server the same panel is also displayed.
URL of GUI (RM server)
user name
password
Figure 35-16 Launch the Replication Manager GUI
Specify a user ID as a text string on the UserID filed and a password in a hidden text field.
User IDs are defined and set up in the RM server system.

35.15.2 Health Overview panel
After a successful log in to the Replication Manager server, the Health Overview panel is
displayed, as shown in Figure 35-17.
Figure 35-17 Health Overview panel
This health panel displayed an overall summary of the Replication Manager system status. It
shows information very similar to what is also shown in the small box in the left lower corner
of this panel. This small health overview box at the lower left corner is always present.
However, the Health Overview panel provides more details. It provides a first overall status of:
򐂰 Sessions
򐂰 Connected storage subsystems
򐂰 Management servers
Figure 35-17 reports that all sessions are in normal status and working fine. There is no high
availability server environment, and one or more storage servers cannot be reached by the
Replication Manager, as they have been defined before.
The upper left box in this panel labeled My Work provides a list of applications that you can
use to manage various aspects of a copy services environment:
򐂰 Health Overview
This is the currently displayed panel, as Figure 35-17 shows.
򐂰 Sessions
This hyperlink brings you to the application that manages all sessions. This is the
application that you will use the most.
򐂰 Storage Subsystems
Here you start when you define storage servers to the RM server that are going to be used
for Copy Services.
򐂰 ESS/DS Paths
This link allows you to manage everything that is related to PPRC path management.
򐂰 Management Servers
This links lead you to the application that manages the RM server configuration.
򐂰 Advanced Tools
Here you may trigger to collect diagnostic information or set the refresh cycle for the
displayed data.
򐂰 Console
This link opens a log that contains all activities that the user did and their results.
35.15.3 Sessions panel

This is the panel to all sessions within the RM server (Figure 35-18).
Figure 35-18 Sessions overview
Each session comprises a number of copy sets that may be distributed across LSSs and
physical storage servers. The session name functions as a token that is used to apply any
action again the session or all the volumes that belong to that session.

Figure 35-19 illustrates that you first select a session and then chose the action you want to
perform against that session.
(2) Select action
(1) Select session

Figure 35-19 About to create copy sets to session
Figure 35-20 displays a next possible step to perform an action. After selecting the session
GM_Session1, as shown in Figure 35-18 on page 565, you may select any action from the
action list shown in Figure 35-20.
Figure 35-20 Actions against a session
This may be an action against the entire session, like suspending all volumes within the
session. It is also used to modify an existing session and add or remove copy sets.
35.15.4 Storage Subsystems panel
The following panel (Figure 35-21) displays all storage subsystems currently connected to the
RM server.
Panel heading
Hyper links
Interface to functional tasks
RM summary
Figure 35-21 GUI basic layout
Using the Add Subsystem button you can define another storage subsystem to the RM
server.
The panel used to add a new server is shown in Figure 35-23 on page 568.

Figure 35-22 displays the available action list. From this list you select, for instance, the
View/Modify Details action and apply it to the previously selected storage server.
Figure 35-22 Select storage subsystem and select View/Modify Details action
The selected storage subsystem connectivity details are now displayed, as shown in
Figure 35-23.
Figure 35-23 Storage subsystem details
You usually use the Storage Subsystems application only to connect a storage server to the
RM server. Figure 35-23 on page 568 also shows the standard port used port used by the RM
server to communicate with the storage server. All the other fields are self explanatory.
35.15.5 Path Management panel

Figure 35-24 displays the entry panel to manage PPRC paths.
Figure 35-24 Path overview panel
Clicking the Manage Path button will trigger the path wizard to help you define a new PPRC
path or remove existing PPRC paths.

Clicking a storage subsystem gives you a list of all the existing paths currently defined for the
selected storage subsystems. Figure 35-25 displays all the defined PPRC paths for the
selected LSS and the ports used for these PPRC paths.
Figure 35-25 Path overview of a DS8000
You may select any path here and the only available action in this case is to then remove the
selected paths.
35.15.6 RM Server Configuration panel

The panel in Figure 35-26 displays the status of the Replication management servers.
Figure 35-26 Replication Management Servers
Figure 35-26 shows only one server named Stops, which is an active server.
When it exists, a second row will show the potential second server, which is in the status
standby. A panel with two servers has a slightly different appearance than what is shown in
Figure 35-26.
Use this panel for basic operations such as defining a server as standby or to take over in the
event of a disaster.
In the case of two servers, each server manages its own DB2 database. The communication
between both servers is performed through the LAN to which both servers are connected.
35.15.7 Advanced Tools panel

Figure 35-27 displays a panel through which you handle some specific tasks.
Figure 35-27 Advanced tools option
Specific tasks in this context are tasks to create a diagnostic package or to change the
automatic refresh rate of the GUI. A third task is to enable or to disable the heartbeat, which
happens between the Replication Manager server and the connected storage servers. Aee
35.12, “TPC for Replication heartbeat” on page 559.
The diagnostic package contains all RM logs and its location on the RM server is shown on
the screen.
The browser refresh rate is currently a value between 5 and 3600 seconds. The default value
is 30 seconds.

35.15.8 Console log
Figure 35-28 displays an example of a console log. This panel shows a list of the most recent
commands, which this user entered through the GUI.
Figure 35-28 Console log
Besides the commands this log shows also whether the command succeeded and a
message number functions at the same time as a hyper link to more detailed text about the
result of the concerned command execution.
35.16 Command line interface to TPC for Replication

Besides the GUI you may also manage TPC for Replication through a command line interface
(CSMCLI).
As with the DSCLI for DS8000 and DSCLI for DS6000, the CSMCLI command structure is
similar between all three CLI products like mk... for make, ch... for change, and.
CSMCLI is also invoked in the same fashion as the DSCLI for the DS products and provides
three different modes:
򐂰 Singe-shot mode
򐂰 Interactive mode
򐂰 Script mode
Example 35-4 shows a single shot command.
Example 35-4 Single-shot CLI command

> csmcli lsdevice -devtype ds
To execute more than one command you may start the CLI program and perform the
commands at the shell prompt, as Example 35-5 shows.
Example 35-5 Interactive CLI commands

... start csmcli ...
csmcli> lsdevice -devtype ds
csmcli> lsdevice -devtype ess
The third mode is the script mode to run commands out of a file.
Example 35-6 Script mode to execute CLI commands

... start csmcli ...
csmcli -script ~/rm/scrtips/devreport
In contrast to DSCLI for the DS storage servers, the CSMCLI does currently not use a -profile
option.

36
Chapter 36. IBM TotalStorage Rapid Data

Recovery for UNIX and Windows
IBM TotalStorage Rapid Data Recovery for Unix and Windows is a service offering solution
from IBM Global Services based on enterprise Remote Copy Management Facility (eRCMF).
eRCMF is a standalone tool that provides a management layer for ESS800, DS8000, and
DS6000 Copy Services in 2-site or 3-site topologies. It is a common management tool for
both z/OS and open system environments. The Copy Services functions managed by
eRCMF are Metro Mirror, Global Copy, and Global Mirror. At each site, a FlashCopy of the
local volumes can also be managed.
This solution simplifies the process of repairing inconsistent Remote Copy pairs after an error
with the copy process. eRCMF assists the user in automating the Remote Copy and
FlashCopy processes.
Attention: Because of its complex implementation, IBM Global Services will work with
your IT department to install and configure the solution to match your unique business
environment. The storage servers, servers, and WebSphere software have to be ordered
through the normal process. IBM Global Services provides eRCMF software and experts
that support the implementation, education, and skill transfer for this solution.

36.1 Introduction
eRCMF is a scalable and flexible solution that protects business data and maintains
consistent data. It can be used to manage copy relationships for both:
򐂰 Planned outages such as hardware and software upgrades, or disaster practice
򐂰 Unplanned outages such as an actual disaster
The solution provides basic commands through the FO/FB Session Manager Proxy, which
utilizes the ESS Open API functions to accomplish either a planned or unplanned FO/FB
sequence. eRCMF commands can be incorporated in any user scripts or utilities that may
further link users’ overall data center operations. However, it is the user’s responsibility for
creation, maintenance, and any liabilities associated with such scripts or utilities. Therefore,
eRCMF offers a simple way to implement Remote Copy disaster recovery procedures via
command line. eRCMF applies the commands to a set of configuration files that describe the
disk subsystem configuration (paths and sets of volumes).
The solution does not depend on the operating system, as no client installation is required.
36.1.1 Solution highlights

The main solution highlights and benefits are summarized in the following list:
򐂰 Provides automation capability
򐂰 Simplifies disaster recovery operations
򐂰 Improves protection of critical business data guaranteeing data consistency and ensuring
restart capabilities
򐂰 Avoids human errors
򐂰 Reduces risk of downtime
򐂰 Stays competitive, maintaining market readiness by managing risk more efficiently
򐂰 Enables test and practice of disaster recovery setups regularly
򐂰 Reduces testing risk
򐂰 Provides a central, scalable, flexible, enterprise class solution that protects your business
򐂰 Simplifies the configuration:
– Simple configuration description
– Provides tools to verify configuration
򐂰 Simplifies the utilization
Simple commands to fix problems
򐂰 No predefined Copy Services tasks necessary
36.2 Overview
In a Business Continuity solution based on disk Remote Mirroring, we want to restart a
database application following an outage without having to restore the database. This
process has to be consistent, repeatable, and fast, measurable in minutes.
A database recovery where you have to restore the last set of Image Copy tapes and apply
the log changes to bring the database up to the point of failure is a process that can be
measured in hours or even days. And this is what you want to avoid in a Tier 4 or later
solution. However, in a real disaster (fire, explosion, earthquake) you can never expect your
complex to fail all at the same moment. Failures will be intermittent and gradual, and the
disaster will occur over many seconds or even minutes. This is known as the Rolling
Disaster. The architecture of a viable disk mirroring disaster recovery solution must avoid
data corruption caused during a rolling disaster.
In any operating system, the sequence in which updates are being made is what maintains
the integrity of the data. If that sequence is changed, data corruption will occur. The correct
sequence must be maintained within a volume, across volumes, and across multiple storage
devices. For instance, in Figure 36-1 we show the relationship between a database and its
log, which demonstrates the requirement for maintaining I/O integrity. Data consistency
across the storage enterprise must be maintained to insure data integrity.
Primary
Primary Secondary
Secondary
Log 1. Log Update
Volume 3. Mark Log Complete
Database
2. Update Database
Volume
Figure 36-1 Sample update sequence in a database
The order of dependent writes across volumes must be maintained at remote locations.
Failure to do so results in data corruption. The data consistency is lost.
Chapter 36. IBM TotalStorage Rapid Data Recovery for UNIX and Windows 577
In Figure 36-2 we illustrate this concept with an example. When a database update is about to
take place, the change is first logged in the database log files at both the primary and
secondary volumes (step 1). Assume the database datafile is updated at the primary volume,
but the update does not reach the remote volume that contains the mirrored data file. The
primary location is not aware of the write failure to the secondary volume (step 2). The
database update is marked complete in the log files at both the primary and remote locations
(step 3). The result is that the secondary site log files say the update was done, but the
updated data is not in the database at the secondary location. There is no way to know that
the data was corrupted.
Primary
Primary Secondary
OK 1. Log Update
B C
OK
L M 3. Mark Log
Complete
X Y
Application
Database
B C
OK 2. Database Update
L M
X Y
Left side disk data

does not match right
side. Is this the
problem?
Yes
Figure 36-2 Sample Rolling Disaster
So, the issue is that the disk subsystem cannot do it all and we must avoid this Rolling
Disaster system problem.
For this reason it is strongly recommended that, at a minimum, Consistency Groups be put in
place for any mirroring solution. Consistency Groups are an implementation of technology
that assists with the consistency of application data capable of dependent writes. To
guarantee a fully consistent remote copy, multiple volumes require a Consistency Group
functionality. ESS and DS6000/DS8000 Remote Copy microcode already have the
Consistency Group function for both z/OS and open systems. SAN Volume Controller Metro
Mirror’s and DS4000 Remote Mirror’s microcode also have a Consistency Group function for
open systems.
If any volume within a Consistency Group is unable to complete a write to its counterpart in
the Remote Copy relationship, an Extended Long Busy (ELB) for mainframe environments or
a SCSI Queue Full condition for open systems will be issued, preventing further writes to any
of the volumes within the Consistency Group. This wait period is the perfect time to issue a
freeze to all volumes involved to maintain consistency. Figure 36-3 on page 579 illustrates
this mechanism. If a write is unable to complete, the storage subsystem will not back out
incomplete transactions on its own. Instead, the application will need to recognize that the
transaction was incomplete and take the appropriate actions. Once the storage subsystem
holds application I/O to the affected primary volumes, the write dependent mechanism of the
application prevents the Remote Copy secondaries from becoming inconsistent.
Consistency Group
control function
SNMP trap
B C
L M
1. Mirroring failure
2. ESS suspends affected primary
X Y
Application
and holds application I/O (SCSI

Database
Queue full condition)

3. ESS sends a notification about
the failure
4. A program like eRCMF can
receive the SNMP trap and issue
B C the freeze to all LSSs in the
Consistency Group
L M 5. ESS releases I/O if told to do so
or after PPRC Consistency
Group time out is over
X Y
Figure 36-3 Dependent write protection of database integrity
freeze is a command available through the DS Command-Line Interface, the DS Storage

Manager (Graphical User Interface), the ESS API, ICKDSF, or TSO that stops the write
activity on all of the active Remote Copy primary and secondary volumes of a given source
and target Logical Subsystem (LSS) pair. This function ensures consistency between the
primary and secondary volumes and can affect any LSS volumes that are in an active
Remote Copy state between the frozen LSS pair: Duplex, duplex pending SYNC, or duplex
pending XD states. It does not affect the suspended and simplex volumes that may be in the
LSS. The freeze operation has three effects:
򐂰 The paths that connect the pair of LSSs being frozen are terminated.
򐂰 The active volumes under the frozen LSS pair are suspended. This state transition, to
suspended, is then communicated to the host with alert messages. These alert messages
can be used by automation routines to trigger other recovery operations.
򐂰 If the Consistency Group option was enabled at path definition time, then the ELB or SCSI
Queue Full condition is instigated, so that the write activity to the primary LSS is
temporarily queued. During this interval other automated operations can be triggered,
such as freezing other application-related LSS pairs.
When using freeze through the DS Storage Manager (Graphical User Interface) or ICKDSF,
it will take effect on each LSS individually. This is useful for creating a Point-in-Time Copy of
the data, but because of slight delays between the issuing of each iteration of the freeze
command it is unsuitable for preventing Rolling Disasters and should be done at periods of
low utilization to ensure that the secondary data can be used.
When Remote Copy is used in conjunction with automation, such as the Geographically
Dispersed Parallel Sysplex™ (GDPS) or enterprise Remote Copy Management Facility
(eRCMF) service offerings from IBM Global Services, a freeze command can be
simultaneously issued to all LSSs within the configuration. This ensures globally consistent
data across all LSSs in the secondary copy of data during a disaster.
The goal of the TotalStorage Rapid Data Recovery for UNIX and Windows solution, based on
the combination of ESS or DS6000 or DS8000 Copy Services with enterprise Remote Copy
Management Facility (eRCMF), is to protect your data from being a mirror of a dying scenario.
Using a Remote Copy Consistency Group, this solution freezes your environment at a known
point instead of mirroring literally hundreds of time-offset failures in a short amount of time.
TotalStorage Rapid Data Recovery is based on enterprise Remote Copy Management

Facility (eRCMF), an IBM Global Services offering. The current eRCMF Version 4 is designed
as a 2-site or 3-site disaster recovery solution that runs on dedicated machines under
WebSphere on AIX 5L™, which will be used as eRCMF servers.
Note: We strongly recommend that the RCMF servers be installed and run from different
physical sites to ensure their backup capability.
eRCMF is a multi-site disaster recovery solution that manages data on volumes for Fixed
Block (open systems hosts) and CKD (z/OS). The solution does not depend on the operating
system, as no client installation is required.
eRCMF is a Tier 4 and 6 solution, as shown in Figure 36-4.
4 Software Automation for SAP, DB2,Siebel,MS SQL Server,

Exchange etc. + di s k
t a pe
3 Automation for Servers: z/OS, UNIX, Linux, Windows and
Autonomic
Business value
heterogeneous environments.
Services
2 Point in Time Copy, Metro Mirror, Global Mirror, TotalStorage

7 6 5 4
Software Family, DFSMS for z/OS, TSM.
3 2 1
1 Hardware Infrastructure: ESS, DS family, SAN Switches, VTS,
3584, 3592, LTO . - Recovery Time
+
Areas covered and tier level
Figure 36-4 eRCMF protection tier
eRCMF is built on the technologies of Remote Copy, Java, WebSphere, and ESS Network
Interface. This is because ESS 800, DS6000, and DS8000 share the same Copy Services
functions that you can mix these storage servers in this solution. With DS8000 and DS6000,
no Copy Services Server is required since each storage server can handle the Copy
Services.
36.3 Architecture
eRCMF is a 2-site or 3-site solution. According to the diagrams in Figure 36-5 on page 581
and Figure 36-6 on page 581, one PCM is placed in each data center. The PCM in the
primary data center runs the master process. This process is the interface into eRCMF. It
handles all queries and commands for local processes through either a command-line or
socket interface. It also manages commands and queries from the Backup Process.
A second PCM can be placed in the secondary (or backup) data center or, in a 3-site
strategy, in the tertiary (or remote) data center. This PCM runs the Backup Process, which
maintains the state information of the configuration. This information is transferred to the
second PCM in the form of logs from the master process. These logs are then used to update
state information in case the backup process must take control of the configuration, as well as
for documentation purposes if there is an alternate site failure.
Metro Mirror, up to:

Site 1 - Production Global Copy, over:
Site 2 - Backup
Storage Server (s) 103 km / 300 km Storage Server (s)
ESCON / FCP
Shadow FlashCopy Host Host FlashCopy Shadow
Metro Mirror, Global Copy
Volumes Volumes Volumes Volumes
Global Mirror
Journal Journal
Volumes Volumes
TCP/IP / SNMP
AIX – Backup PCM AIX – Primary PCM

WebSphere - HTTPS WebSphere - HTTPS
ESS Network Interface ESS Network Interface
eRCMF eRCMF
eRCMFconfig.properties eRCMFconfig.properties
Enterprise.dat Enterprise.dat
VolumeSet.dat VolumeSet.dat
Hosts.dat Hosts.dat
Authentication.file Authentication.file
Backup process Master process

(Master process) Management SNMP (Backup process)
Interface Listener
Figure 36-5 eRCMF architecture: 2-site configuration
Metro Mirror, up to Global Copy, over

103 km / 300 km 103 km / 300 km
Site 1 - Production Site 2 - Backup Site 3 - Remote

Storage Server(s) ESCON Storage Server(s) ESCON
Storage Server(s)
FCP FCP
Shadow Host Host Shadow Host Shadow
Volumes FlashCopy Volumes Metro Volumes FlashCopyVolumes Global Volumes FlashCopyVolumes
Mirror Copy
Copy Services Copy Services Copy Services
(CSServer on ESS 800 / 750) (CSServer on ESS 800 / 750)
AIX – Backup PCM TCP/IP AIX – Primary PCM

WebSphere - HTTPS SNMP WebSphere - HTTPS
ESS Network Interface ESS Network Interface
eRCMF eRCMF
eRCMFconfig.properties eRCMFconfig.properties
Enterprise.dat Enterprise.dat
VolumeSet.dat VolumeSet.dat
Hosts.dat Hosts.dat
Authentication.file Authentication.file
Backup process Master process

(Master process) Management SNMP (Backup process)
Interface Listener
Figure 36-6 eRCMF architecture: 3-site configuration
The master process continuously monitors and manages the entire environment at two
levels:
򐂰 Enterprise level
򐂰 VolumeSet level
The distinction between these levels is primarily the scope of actions performed at each level.
The Enterprise level represents the entire monitored environment, which consists of two or
three data processing locations containing storage managed by eRCMF. The VolumeSet
level corresponds to a set of storage server volumes, which must be maintained in a
consistent state and managed using the same type of copy operations. Some examples of a
VolumeSet are the volumes used by a set of applications, the volumes allocated to a set of
servers, or the members of a Volume Group.
Management at the Enterprise level affects the entire monitored environment. These actions
typically affect every volume of every VolumeSet being managed by eRCMF. For example,
the freeze command will cause suspension of Remote Copy pairs between the storage
subsystems on multiple sites, as you can see in Figure 36-7.
eRCMF adds a central control function to coordinate the Remote Copy Freeze function and
stops transmission of data to the remote site consistently across single/multiple frames.
When this function is centrally coordinated, data is consistent. Data consistency allows a very
fast, very consistent restart time at the secondary site.
yNotification of mirroring
error
•eRCMF issues freeze command to all Storage
yCU holds I/Os to eRCMF Servers
suspended device until •suspends all PPRC pairs
eRCMF can handle •holds I/O until told to continue
condition
•Based on policy, eRCMF will then allow host
operation to resume or continue to hold them
Control Control
Host Unit
Control Unit
Control
During
Host Unit Unit
Host
Control Control
Unit
Control Unit
Control
Host
Unit Unit
Control Control
Control
Unit Control
Unit After, depending on policy:
Unit
Control Unit
Control
Unit
Control Unit
Control •Both sites are suspend but in sync
Unit Unit
•Sites are not insync, however data
to a site is consistent
Data at remote site is at a “Known State”

Figure 36-7 eRCMF issues freeze command in a disaster recovery scenario
There are two methods of maintaining consistent data:

򐂰 Metro Mirror
– Use Synchronous Remote Copy to ensure that data is secure at the remote site before
signaling the host that the write is complete.
– During a disaster, combine the Consistency Group and Freeze functions in the storage
server with software, such as eRCMF, to insure that all Remote Copy pairs of the
Consistency Group are split in a manner that assures data consistency at the remote
site.
򐂰 Global Mirror
– The storage server periodically forms Consistency Groups.
– When recovering to the recovery copy, software such as eRCMF examines the states
of the Global Mirror relationships and directs the storage server through the steps
necessary to recover that data to the last committed consistent copy.
It should be noted that eRCMF neither performs an automated recovery nor an automated
restart of systems in the remote site. Instead, eRCMF's freeze performs a controlled site
split only. If a higher level of automation is required, eRCMF provides interfaces such as
execution scripts or a command-line interface, which can be utilized and customized. When
this occurs, mirroring between site 1 and site 2 stops, ensuring the consistency of data.
Further response by the automation is determined by policies that are selected based on
individual business requirements:
򐂰 Freeze and Go quickly executes the freeze to ensure data consistency, but then
immediately allows I/Os to continue. This allows processing to continue in the production
data center while maintaining a consistent copy of the data in either the backup or remote
data center. Meanwhile, the business is able to determine the cause of the event and
whether it is a disaster. If a disaster has not occurred, production has not been impacted
by an unnecessary site switch. If a disaster has occurred, then the data since the time of
the site split may need to be recreated if recovering to a backup or a remote data center.
򐂰 Freeze and Stop quickly executes the freeze to ensure consistency but, rather than
immediately allowing the I/Os to continue, causes the storage server to block all further
I/Os until told otherwise. This ensures that no transactions, other than those in-flight at the
time of the event, need to be recreated following a disaster. If the event turns out not to be
a disaster, however, the production will be halted until the I/Os are freed.
36.4 Additional information

Further information can be found on the IBM Web site at:
http://www-1.ibm.com/services/us/index.wss/so/its/a1000110
For further information you can refer to the following papers:

򐂰 enterprise Remote Copy Management Facility eRCMF V4 Implementation Guide
򐂰 enterprise Remote Copy Management Facility eRCMF V4 User Guide
You can find these at:

http://www-1.ibm.com/support/docview.wss?uid=ssg1S7001074
For more information contact your IBM sales representative.
37
Chapter 37. IBM TotalStorage Continuous

Availability for Windows
IBM TotalStorage Continuous Availability for Windows (formerly named IBM TotalStorage
Geographically Dispersed Clusters for Microsoft® Cluster Server, or GDS offering) on the
Enterprise Storage Server (ESS), DS6000, DS8000, and SAN Volume Controller is designed
to allow cluster resources (that is, applications and data) to fail over between two
geographically dispersed sites. By combining the high availability of Microsoft Cluster Server
(MSCS) services for application cluster failover with Metro Mirror, TotalStorage Continuous
Availability for Windows provides a continuous availability solution to protect critical
cluster-aware applications and data in a Microsoft Cluster environment.
This services offering, available by RPQ through IBM Systems Group, enables high
availability and disaster recovery in the Microsoft environment by incorporating the benefits of
Metro Mirror.

37.1 Introduction
IBM TotalStorage Continuous Availability for Windows is the IBM implementation for disk
subsystems within the Microsoft Geographically Dispersed Sites for clusters framework. It will
provide clustering failover support for any MSCS cluster-aware applications, such as DBMS
messaging and file sharing. Any MSCS cluster-aware applications will run in a TotalStorage
Continuous Availability for Windows configuration.
IBM TotalStorage Continuous Availability for Windows is a disaster recovery solution for
Microsoft Cluster server applications using automated cluster failover services combined with
IBM storage subsystems using Metro Mirror for remote site mirroring and automated failover
of data. It can allow for zero or little data loss and little or no downtime, and it provides for
complete site server application and data disaster recovery across two geographically
dispersed sites.
It monitors the health states of the applications, network, and Metro Mirror storage to
determine automatic actions for failover to the secondary site.
Solution highlights
The main solution highlights and benefits are summarized in the following list:
򐂰 Improve continuity of business.
򐂰 Improve protection of critical business data.
򐂰 Reduce business risk.
򐂰 Provide flexibility for the transfer of operations between sites with minimal disruption.
򐂰 Improve business resiliency.
򐂰 Stay competitive, maintaining market readiness by managing risk more efficiently.
37.2 Overview
When you are using Microsoft Clustering Services with Windows 2003 Enterprise Edition
Servers, you may not only want to prevent your business from a disaster on a single node
participating in the cluster but might also want to prevent a disaster on the disk subsystems
shared by the nodes. The solution is to have a Geographically Dispersed Solution where both
servers and storages resources are redundant and installed on different sites. In such a case,
you need to mirror your shared disks between both sites. Since you cannot use any RAID
controller card in a SAN environment, two choices are possible. Either you ask the Windows
system to do it or you ask the disk subsystem to do it.
The first solution is based on a system mirror. When you use a system mirror, the
synchronization process is supported by the resources of your cluster server node. Because
Microsoft Clustering Services does not support dynamic disks, the only way to perform it is to
add a software layer between your system and the disks. This kind of layer not only adds a
new level of complexity to your system and to your storage management, but it also has some
distance limitations between the two sites.
The second solution is based on a disk subsystem mirror such as the DS8000 Metro Mirror.
When you use a disk subsystem mirror, the synchronization process is supported by the
resources of your disk subsystem and therefore is not impacting the resources of your cluster
server nodes. At the same time, you can take advantage of the replication technology
installed on the disk subsystems, which allows replication at greater distances. It is a way to
let every actor in this clustering solution do its job. The cluster node is computing while the
disk subsystem is replicating.
When you choose the second solution, the challenge is to automate the failover and failback
process between remote sites involving not only cluster nodes but also disk subsystems. With
IBM TotalStorage Continuous Availability for Windows you can perform such tasks
transparently from your cluster management interface.
IBM TotalStorage Continuous Availability for Windows (formerly Geographically Dispersed

Sites for Microsoft Cluster Services) integrates Microsoft Cluster Services and the Metro
Mirror Advanced Copy Service functions of the IBM TotalStorage DS8000. This product is
implemented on Windows 2003 Enterprise Edition using two sites. Each pair of disks
belonging to the same cluster resource group can only be assigned to a pair of cluster nodes
(one on each site). Therefore this solution allows either Active/Passive or Active/Active cluster
architectures. Of course, at a clustered resource group level there is only Active/Passive
solutions since you cannot write concurrently on two volumes of a Metro Mirror relation.
If you choose the Active/Active cluster architecture, it means that every node at both sites can
be actively running applications. There are active resources groups on sites A and B. Your
production is naturally balanced over both nodes sharing these resource groups. When a
resource group on site A has a failure, the data and applications bound to this resource group
automatically fail over to site B. Depending on your application scripts, it can take a few
seconds or several minutes. This architecture has two advantages. First, you can be sure that
each node hosting a resource group is in a good state, and therefore not discover a wrong
behavior when it is too late (such as during a cluster failover following a disaster). Second,
you can balance your production’s workload on both nodes and therefore get better
performance. You can encounter some performance issues during a disaster scenario if the
remaining node is not able to handle the full production workload.
If you choose the Active/Passive cluster architecture, when the primary site suffers a failure,
the data and applications automatically fail over to the backup site, typically in minutes. Once
the primary site is repaired, the resources can be failed back to the original site.
Note: A failover/failback process at a cluster level is not similar to a failover/failback at a

Metro Mirror level.
For a cluster, a failover process is when the production is moving from a site A to a site B.
A failback is when the production is moving back from a site B to a site A.
For Metro Mirror, a failover is when the replication process is stopped and the production
volume is moved from site A to site B. At that time both volumes participating in the former
pair are accessible. A failback is when after a failover the replication process is reactivated
with the source volume on site B and the target volume on site A.
Chapter 37. IBM TotalStorage Continuous Availability for Windows 587

IBM TotalStorage Continuous Availability for Windows offers high availability for servers and
applications plus disaster recovery for files and data. It improves data availability with
continued service during hardware and software failures. It provides disaster-tolerant
capabilities by allowing the cluster servers and associated mirrored storage to be
geographically separated by the Metro Mirror distance for FCP. It can also be used to reduce
or eliminate planned downtime, such as during a software or hardware upgrade. GDS is a
Tier 7 High Availability solution. We can see the topology in Figure 37-1.
4 Software Automation for SAP, DB2,Siebel,MS SQL Server,

Exchange etc. + disk
tape
3 Automation for Servers: z/OS, UNIX, Linux, Windows and
Business value
Autonomic
heterogeneous environments.
Services
2 Point in Time Copy, Metro Mirror, Global Mirror, TotalStorage

Software Family, DFSMS for z/OS, TSM. 7 6 5 4 3 2
1
1 Hardware Infrastructure: ESS, DS family, SAN Switches, VTS,
3584, 3592, LTO . - Recovery Time
+
Areas covered and tier level
Figure 37-1 TotalStorage Continuous Availability for Windows protection tier
Because the ESS 800, DS6000, and DS8000 share the same Copy Services functions you
can mix them in this solution, ESS 800 or DS6000 or DS8000. The disk subsystems are
connected to each other using PPRC links, and the host servers are fibre attached to the disk
subsystems at each site. The cluster-shared disks are mirrored between two disk
subsystems. The solution uses PPRC to coordinate the ownership and movement of disk
resources between sites.
37.3 Architecture
IBM TotalStorage Continuous Availability for Windows is a 2-site architecture with two MSCS
cluster nodes. Each client of the cluster’s resources are connected to the cluster through an
Ethernet LAN or WAN connection. For availability reasons, at least one other Ethernet
connection is linking both nodes for heartbeat purposes. The reason for this double TCP/IP
connection is that if any of this network should fall, each node could use the secondary
network to communicate, and therefore avoid an unnecessary failover.
Replication PPRC links must be defined in both directions in order for the replication pairs of
volumes to be able to fail over and fail back automatically and attempt to always replicate the
writes on the other site.
A quorum volume should be created on each site and a Metro Mirror relation should be
established between each of them. Then each volume composing a cluster resource group
should be created on both sites and a Metro Mirror relationship should be established
between each of them. Finally, for the IBM TotalStorage Continuous Availability for Windows
solution to be cluster aware, two pairs of so-called site disks should be created, and a cross
Metro Mirror relation should be established between each of them. It means that one pair will
replicate from site A to site B and the other pair from site B to site A. These Metro Mirror
relations will never be switched in the future.
Figure 37-2 Architecture of the solution
37.3.1 Service and ordering

Because of its complex implementation in a clustered environment, IBM TotalStorage
Continuous Availability for Windows is provided as an IBM System Group Services offering.
The offering includes three components:
򐂰 TotalStorage Continuous Availability for Windows software license, which includes C++
code and an Install package for installation and configuration within an MSCS environment
across two nodes.
򐂰 A 5-day on-site services engagement to help the customer set up, configure, skill transfer
on this solution, and validate the installation.
򐂰 An IBM 7x24 Support contract.
Being a high-availability mission-critical disaster recovery infrastructure, TotalStorage

Continuous Availability for Windows is supported by IBM 7x24 Support. The clients can count
on IBM to provide a single point of support focus and effectively work jointly with Microsoft to
support complex customer configurations.
Chapter 37. IBM TotalStorage Continuous Availability for Windows 589

IBM TotalStorage Continuous Availability for Windows has been generally available since
2003 and has been put into production use by clients worldwide. You can order TotalStorage
Continuous Availability for Windows automation code and services through IBM TotalStorage
HUB (RPQ) via your IBM Sales Representative. The storage subsystems and servers have to
be ordered through the normal process.
Additional scalability of extending the current 2-node cluster configuration to 4 and 8 nodes
(the maximum number of cluster nodes presently supported by Windows 2003 Enterprise
Edition) is planned for a future TotalStorage Continuous Availability for Windows release.
Additional information can be found on the IBM Web site at:

http://www-03.ibm.com/servers/storage/solutions/business_continuity/continuous_availability
/technical_details.html
For more information contact your IBM sales representative.
A
Appendix A. Open systems specifics

In this appendix we describe the basic tasks that need to be performed on the individual host
systems when using the DS Copy Services.
We explain how to bring FlashCopy target volumes online to the same host as well as to a
second host. This appendix covers various UNIX and Windows platforms.

AIX specifics
In this section we describe the steps needed to use volumes created by the DS Copy
Services on AIX hosts.
AIX and FlashCopy

The FlashCopy functionality is to copy the entire contents of a source volume to a target
volume. If the source volume is defined to the AIX Logical Volume Manager (LVM), all of its
data structures and identifiers are copied to the target volume as well. This includes the
Volume Group Descriptor Area (VGDA), which contains the Physical Volume Identifier (PVID)
and Volume Group Identifier (VGID).
For AIX LVM, it is currently not possible to activate a Volume Group with a physical volume
(hdisk or vpath) that contains a VGID and a PVID that is already used in a Volume Group
existing on the same server. The restriction still applies even if the hdisk PVID is cleared and
reassigned with the two commands listed in Example A-1.
Example: A-1 Clearing PVIDs

#chdev -l <hdisk#> -a pv=clear
#chdev -l <hdisk#> -a pv=yes
Therefore, it is necessary to redefine the Volume Group information about the FlashCopy
target volumes using special procedures or the recreatevg command. This will alter the
PVIDs and VGIDs in all the VGDAs of the FlashCopy target volumes, so that there are no
conflicts with existing PVIDs and VGIDs on existing Volume Groups that reside on the source
volumes. If you do not redefine the Volume Group information prior to importing the Volume
Group, then the importvg command will fail.
Accessing FlashCopy target volume from another AIX host

The following procedure makes the data of the FlashCopy target volume available to another
AIX host that has no prior definitions of the target volume in its configuration database (ODM).
This host that is receiving the FlashCopy target volumes can manage the access to these
devices in two different ways:
򐂰 LVM or MPIO definitions
򐂰 SDD definitions
If the host is using LVM or MPIO definitions that work with hdisks only, follow these steps:
1. The target volume (hdisk) is new to AIX, and therefore the Configuration Manager should
be run on the specific Fibre Channel adapter:
#cfgmgr -l <fcs#>
2. Find out which of the physical volumes is your FlashCopy target volume:
#lsdev -C |grep 2107
3. Certify that all PVIDs in all hdisks that will belong to the new Volume Group were set.
Check this information using the lspv command. If they were not set, run the following
command for each one to avoid the importvg command failing:
4. Import the target Volume Group:
#importvg -y <volume_group_name> <hdisk#>
5. Vary on the Volume Group (the importvg command should vary on the Volume Group):
#varyonvg <volume_group_name>
6. Verify consistency of all file systems on the FlashCopy target volumes:
#fsck -y <filesystem_name>
7. Mount all the target file systems:
#mount <filesystem_name>
If the host is using SDD that works with vpath devices, follow these steps:
1. The target volume (hdisk) is new to AIX, and therefore the Configuration Manager should
be run on all Fibre Channel adapters:
#cfgmgr -l <fcs#>
2. Configure the SDD devices:
#smitty datapath_cfgall
3. Find out which of the physical volumes is your FlashCopy target volume:
#lsdev -C | grep 2107
4. Certify that all PVIDs in all hdisks that will belong to the new Volume Group were set.
Check this information using the lspv command. If they were not set, run the following
command on each one to avoid the importvg command failing:
#importvg -y <volume_group_name> <hdisk#>
6. Change the ODM definitions to work with the SDD devices that will provide you with the
load balance and failover functions:
#dpovgfix <volume_group_name>
7. Vary on the Volume Group (the importvg command should vary on the Volume Group):
#varyonvg <volume_group_name>
8. Verify the consistency of all file systems on the FlashCopy target volumes:
#fsck -y <filesystem_name>
#mount <filesystem_name>
The data is now available. You can, for example, back up the data residing on the FlashCopy
target volume to a tape device. This procedure can be run once the relationship between the
FlashCopy source and target volume is established, even if data is still being copied in the
background.
The disks containing the target volumes may have been previously defined to an AIX system,
for example, if you periodically create backups using the same set of volumes. In this case,
there are two possible scenarios:
򐂰 If no Volume Group, file system, or logical volume structure changes were made, then use
procedure 1 (“Procedure 1” on page 594) to access the FlashCopy target volumes from
the target system.
򐂰 If some modifications to the structure of the Volume Group were made, such as changing
the file system size or the modification of logical volumes (LV), then it is recommended to
use procedure 2 (“Procedure 2” on page 594) and not procedure 1.
Appendix A. Open systems specifics 593

Procedure 1
The procedure is:
1. Unmount all the source file systems:
#umount <source_filesystem>
2. Unmount all the target file systems:
#umount <target_filesystem>
3. Deactivate the target Volume Group:
#varyoffvg <target_volume_group_name>
4. Establish the FlashCopy relationships.
5. Mount all the source file systems:
#mount <source_filesystem>
6. Activate the target Volume Group:
#varyonvg <target_volume_group_name>
7. Perform a file system consistency check on target file systems:
#fsck -y <target_file_system_name>
#mount <target_filesystem>
Procedure 2
The procedure is:
1. Unmount all the target file systems:
#umount <target_filesystem>
2. Deactivate the target Volume Group:
#varyoffvg <target_volume_group_name>
3. Export the target Volume Group:
#exportvg <target_volume_group_name>
4. Establish the FlashCopy relationships.
#importvg -y <target_volume_group_name> <hdisk#>
6. Perform a file system consistency check on target file systems:
#mount <target_filesystem>
Accessing the FlashCopy target volume from the same AIX host
In this section we describe a method of accessing the FlashCopy target volume on a single
AIX host while the source volume is still active on the same server. The procedure is intended
to be used as a guide and may not cover all scenarios.
If you are using the same host to work with source and target volumes, you have to use the
recreatevg command.
The recreatevg command overcomes the problem of duplicated LVM data structures and
identifiers caused by a disk duplication process such as FlashCopy. It is used to recreate an
AIX Volume Group (VG) on a set of target volumes that are copied from a set of source
volumes belonging to a specific VG. The command will allocate new physical volume
identifiers (PVIDs) for the member disks and a new Volume Group identifier (VGID) to the
Volume Group. The command also provides options to rename the logical volumes with a
prefix you specify, and options to rename labels to specify different mount points for file
systems.
Accessing FlashCopy target volume using the recreatevg command

In this example, we have a Volume Group containing two physical volumes (hdisks) and wish
to FlashCopy the volumes for the purpose of creating a backup.
The source volume group is src_flash_vg, containing vpath2 and vpath3.
The target volume group will be tgt_flash_vg, containing vpath4 and vpath5.
Perform these tasks to run the FlashCopy and make the target volumes available to AIX:
1. Stop all I/O activities and applications that access the FlashCopy source volumes.
2. Establish the FlashCopy pairs with the No Background Copy option selected. Use the
DS Storage Manager to establish the pairs or run a script with the DS CLI command.
3. Restart applications that access the FlashCopy source volumes.
4. The target volumes, vpath4 and vpath5, will now have the same volume group data
structures as the source volumes vpath2 and vpath3. Clear the PVIDs from the target
hdisks to allow a new Volume Group to be made:
#chdev -l vpath4 -a pv=clear
#chdev -l vpath5 -a pv=clear
The output of lspv command shows the result in Figure A-1.
Figure A-1 lspv output before recreating the volume group
5. Create the target volume group and prefix all file system path names with /backup, and
prefix all AIX logical volumes with bkup:
recreatevg -y tgt_flash_vg -L /backup -Y bkup vpath4 vpath5
You must specify the hdisk names of all disk volumes participating in the volume group.
The output from lspv shown in Figure A-2 illustrates the new volume group definition.
Figure A-2 lspv output after recreating the volume group
An extract from /etc/filesystems in Figure A-3 on page 596 shows how recreatevg
generates a new file system stanza. The file system named /prodfs in the source Volume
Group is renamed to /bkp/prodfs in the target volume group. Also, the directory
/bkp/prodfs is created. Notice also that the logical volume and JFS log logical volume have
been renamed. The remainder of the stanza is the same as the stanza for /prodfs.

Figure A-3 Target file system stanza
6. Perform a file system consistency check for all target file systems:
7. Mount the new file systems belonging to the target volume group to make them
accessible.
AIX and Remote Mirror and Copy

When you have the primary and secondary volumes in a Remote Mirror and Copy
relationship, it is not possible to read the secondary unless the “Permit read access from
target” and “Reset Reserve” options have been selected when establishing the relationship.
To be able to read the secondary volumes, they must also be in the full duplex state (in
addition to the “Permit read access from target” and “Reset Reserve” options). Therefore, if
you are configuring the secondary volumes on the target server, it is necessary to terminate
the copy pair relationship. Once the volumes are in the simplex state, the secondary volumes
can be configured (cfgmgr) into the target system’s customized device class (CuDv) of the
ODM. This will bring in the secondary volumes as hdisks and will contain the same physical
volume IDs (PVID) as the primary volumes. Because these volumes are new to the system,
there is no conflict with existing PVIDs. The Volume Group on the secondary volumes
containing the logical volume (LV) and file system information can now be imported into the
Object Data Manager (ODM) and the /etc/filesystems file using the importvg command.
If the secondary volumes were previously defined on the target AIX system as hdisks or
vpaths, but the original Volume Group was removed from the primary volumes, the old
volume group and disk definitions must be removed (exportvg and rmdev) from the target
volumes and redefined (cfgmgr) before running importvg again to get the new volume group
definitions. If this is not done first, importvg will import the volume group improperly. The
volume group data structures (PVIDs and VGID) in ODM will differ from the data structures in
the VGDAs and disk volume super blocks. The file systems will not be accessible.
If the secondary volumes that are already configured on the target AIX server are in a
Remote Mirror and Copy relationship and you do not have the “Permit read access from
target” and “Reset Reserve” options enabled (and the volumes are not in full duplex state),
after rebooting the target server, the hdisks will be configured to AIX again. In other words,
you will see each Remote Mirror and Copy secondary volume twice on the target server. The
reason for this situation is as follows: AIX knows that these physical volumes already exist
with entries in the Configuration Database (ODM). However, when the configuration manager
runs during reboot, it cannot read their PVIDs because, as Remote Mirror and Copy targets,
they are locked by the DS Copy Services server. This results in AIX causing the original
hdisks to be configured to a Defined state, and new (phantom) hdisks being configured and
placed in an Available state. This is an undesirable condition that must be remedied before
the secondary volumes can be accessed.
To access the secondary volumes, the phantom hdisks must be removed and the real or
original hdisks must be changed from a Defined state to an Available state. For example,
vpath2 through vpath5 are assigned to a volume group, tgtvg. Each of the disk volumes is
currently participating as a secondary volume in a Remote Mirror and Copy relationship. If the
server is rebooted, four new vpaths are configured to AIX. These phantom disks, vpath6
through vpath9, appear in the output from lspv, as shown in Figure A-4.
Figure A-4 Remote Mirror and Copy phantom disks
Making updates to the LVM information

When performing Remote Mirror and Copy between primary and secondary volumes, the
primary AIX host may create/modify or delete existing LVM information from a Volume Group.
However, because the secondary volume is not accessible when in a Remote Mirror and
Copy relationship, the LVM information in the secondary AIX host would be out-of-date.
Therefore, scheduled periods need to be allotted where write I/Os to the primary Remote
Mirror and Copy volume can be quiesced and file systems unmounted. At this point, the copy
pair relationship can be terminated and the secondary AIX host can perform a learn on the
volume group (importvg -L).
Once the updates have been imported into the secondary AIX host’s ODM, you can establish
the Remote Mirror and Copy pair again. However, select Do not copy volume from the
Select copy Options when establishing the Remote Mirror and Copy pair. As soon as the
Remote Mirror and Copy pair has been established, immediately suspend the Remote Mirror
and Copy relationship. Because there was no write I/O to the primary volumes, both the
primary and secondary are consistent.
Now that the primary volume has been suspended, the file systems can be remounted and
write I/O resumed. Once the write I/O has been going for a while, you can reestablish the
relationship with the primary and secondary by choosing Copy out-of-sync cylinders only.
If the “Permit read access from target” and “Reset Reserve” options were selected during the
Remote Mirror and Copy copy pair establish, then it would be advisable to suspend the
primary volume (while in the full duplex state) and then perform the import learn function.
Once completed, all that is necessary would be to reestablish the copy pair by only copying
the out-of-sync cylinders.
The following example shows two systems, host1 and host2, where host1 has the primary
volume vpath5 and host2 has the secondary volume vpath16. Both systems have had their
ODMs populated with the volume group itsovg from their respective Remote Mirror and Copy
volumes and, prior to any modifications, both systems’ ODM have the same time stamp, as
shown in Figure A-5.
Figure A-5 Original time stamp
Volumes vpath5 and vpath16 are in the Remote Mirror and Copy duplex state, and the
volume group itsovg on host1 is updated with a new logical volume. The time stamp on the

VGDA of the volumes gets updated and so does the ODM on host1, but not on host2. See
Figure A-6.
Figure A-6 Update source time stamp
To update the ODM on the secondary server, it is advisable to suspend the Remote Mirror
and Copy pair prior to performing the importvg -L command to avoid any conflicts from LVM
actions occurring on the primary server. Figure Figure A-7 shows the updated ODM entry on
host2.
Figure A-7 Update secondary server ODM
Once the importvg -L command has completed, you can reestablish the Remote Mirror and
Copy pairs and copy only the out-of-sync cylinders.
Windows and Remote Mirror and Copy

Windows 2000 and 2003 handle their disks differently than Windows NT® does. They
incorporate a stripped-down version of the VERITAS Volume Manager, called the Logical Disk
Manager (LDM).
With the LDM, you are able to create logical partitions, perform disk mounts, and create
dynamic volumes. There are five types of dynamic volumes: Simple, spanned, mirrored,
striped, and RAID 5.
On Windows NT, the information relating to the disks was stored in the Windows NT registry.
With Windows 2000 and 2003, this information is stored on the disk drive itself in a partition
called the LDM database, which is kept on the last few tracks of the disk. Each volume has its
own 128-bit Globally Unique Identifier (GUID) and belongs to a disk group. This is similar to
the concept of Physical Volume Identifier (PVID) and Volume Group in AIX. As the LDM is
stored on the physical drive itself, with Windows 2000 it is possible to move disk drives
between different computers.
Copy Services limitations with Windows 2000 and Windows 2003
Having the drive information stored on the disk itself imposes some limitations when using
Copy Services functionality on a Windows system:
򐂰 The source and target volumes must be of the same physical size. Normally the target
volume can be bigger than the source volume; with Windows, this is not the case, for two
reasons:
– The LDM database holds information relating to the size of the volume. As this is
copied from the source to the target, if the target volume is a different size from the
source, then the database information will be incorrect, and the host system will return
an exception.
– The LDM database is stored at the end of the volume. The copy process is a
track-by-track copy; unless the target is an identical size to the source, the database
will not be at the end of the target volume.
򐂰 It is not possible to have the source and target FlashCopy volumes on the same Windows
system when they were created as Windows dynamic volumes. The reason is that each
dynamic volume has to have its own 128-bit GUID. As its name implies, the GUID must be
unique on one system. When you perform FlashCopy, the GUID gets copied as well, so
this means that if you tried to mount the source and target volume on the same host
system, you would have two volumes with exactly the same GUID. This is not allowed,
and you will not be able to mount the target volume.
Copy services with Windows volumes

In order to see target volumes on a second Windows host, you have to do the following:
1. Perform the Remote Mirror and Copy/FlashCopy function onto the target volume. Ensure
that when using Remote Mirror and Copy that the primary and secondary volumes were in
duplex mode, and write I/O was ceased prior to terminating the copy pair relationship.
2. Reboot the host machine on which you wish to mount the Copy Services target volume.
3. Right-click Open Computer Management, and then click Disk Management.
4. Find the disk that is associated with your volume. There are two panes for each disk; the
left one should read Dynamic and Foreign. It is likely that no drive letter will be associated
with that volume.
5. Right-click that pane and select Import Foreign Disks. Select OK, then OK again. The
volume now has a drive letter assigned to it, and is of Simple Layout and Dynamic Type.
You can read/write to that volume.
Note: When using Windows dynamic disks, remember that to read FlashCopy targets, if
the FlashCopy pair has been rescanned or if the server reading targets has been rebooted,
FlashCopy targets will appear as foreign disks to that server. Manual intervention will be
required to re-import those disks and restore operation.
Tip: Disable the Fast-indexing option on the source disk; otherwise, operations to that
volume get cached to speed up disk access. However, this means that data is not flushed
from memory and the target disk may have copies of files/folders that were deleted from
the source system.
When performing subsequent Remote Mirror and Copy/FlashCopies to the target volume, it is
not necessary to perform a reboot, because the target volume is still known to the target
system. However, in order to detect any changes to the contents of the target volume, you

should remove the drive letter from the target volume before doing the FlashCopy. Then, after
carrying out the FlashCopy, you restore the drive letter in order for the host it is mounted on to
be able to read/write to it.
There is a Windows utility, DiskPart, that enables you to script these operations so that
FlashCopy can be carried out as part of an automated backup procedure. DiskPart can be
found at the Microsoft download site:
http://www.microsoft.com/downloads
with a search on the key word DiskPart. A description of DiskPart commands can be found at
the Web site:
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/diskpart.mspx
Extending simple volumes

The Copy Services source may initially be a single simple volume. However, as requirements
change on the application server, the logical volume may be extended over two or more
volumes. However, you should not independently extend the target volumes, but let Windows
detect the correct sequence of the extended volumes during the import process. For this
reason, the target volumes should be reverted back to basic disks prior to the initial
FlashCopy after the source has been extended. The target server should also be rebooted for
disk manager to pick up the new volumes.
After reboot, the volumes will be recognized as foreign disks, and you can proceed to import
them. Reboot of the target system on subsequent FlashCopy is not necessary until the
source volume has been further extended. When performing subsequent Remote Mirror and
Copy/FlashCopy to the target volume, it is not necessary to perform a reboot because the
target volume is still known to the target system. However, in order to detect any changes to
the contents of the target volume, you should remove the drive letter from the target volume
before doing the FlashCopy. Then, after carrying out the FlashCopy, you restore the drive
letter in order for the host it is mounted on to be able to read/write to it.
There is a Windows utility, DiskPart, that enables you to script these operations so that
FlashCopy can be carried out as part of an automated backup procedure. DiskPart can be
found at the Microsoft download site with a search for the key word DiskPart.:
http://www.microsoft.com/downloads
The DiskPart tool also provides a way to extend an existing partition into free space at the
end of the same logical drive. A description of this procedure can be found in the Microsoft
Knowledge Base, article 304736:
http://support.microsoft.com/?kbid=304736
Enlarging extended/spanned volumes

When you have extended or spanned disks, the logical drive may in time grow to include
more of the initial volume (extended disk) or include additional volumes. When this occurs, it
is necessary, as before, to remove the target volume group information and revert the target
volumes back to basic disks. On the initial FlashCopy, it is necessary to reboot the target
server to configure the additional disks, and then import all the foreign disks that are part of
the volume group.
When performing subsequent Remote Mirror and Copy/FlashCopy to the target volume, it is
not necessary to perform a reboot, because the target volume is still known to the target
system. However, in order to detect any changes to the contents of the target volume, you
should remove the drive letter from the target volume before doing the FlashCopy. Then, after
carrying out the FlashCopy, you restore the drive letter in order for the host it is mounted on to
be able to read/write to it.
Again, we refer to the Windows utility DiskPart, which enables you to script these operations
so that FlashCopy can be carried out as part of an automated backup procedure. DiskPart
can be found at the Microsoft download site:
http://www.microsoft.com/downloads;
Search for the key word DiskPart.
Microsoft Volume Shadow Copy Services (VSS)

The Microsoft Volume Shadow Copy Services (VSS) is a storage management interface for
Microsoft Windows Server® 2003. VSS enables your storage array to interact with third-party
applications that use the VSS Application Programming Interface (API). Microsoft VSS is
included in the Windows Server 2003 installation.
In Windows 2000, each storage area network (SAN) hardware vendor provided its own
proprietary set of APIs for managing their hardware. This makes it challenging to develop
uniform SAN-management software. Windows Server 2003 came out with Volume Shadow
Copy Service, because it is a general infrastructure for creating consistent point-in-time
copies of data on a volume. They are generally referred to as Shadow Copies. VSS is used
for a number of purposes such as:
򐂰 Creating consistent backups of open files and applications
򐂰 Creating shadow copies for shared folders
򐂰 For backup, testing, and data mining
The DS8000 provides an integration with Microsoft Volume Shadow Copy Service to produce
consistent shadow copies.
The IBM System Storage DS8000 Series has FlashCopy functionality, which can integrate
with the Microsoft Virtual Shadow Copy Service. DS8000 FlashCopy enables you to create
full volume copies of data in a Storage Unit. During FlashCopy operation on DS8000, it takes
a few seconds to complete the process of establishing the FlashCopy pair and creating the
necessary control bitmaps.
Volume Shadow Copy Services product components

Prior to Microsoft VSS, if you did not have an online backup solution implemented, you either
had to stop activities on your server during the Backup Process, or live with the side effects of
an online backup, such as inconsistent data and open files that could not be backed up. With
Windows Server 2003 and VSS enabled applications, online backup results in consistent
data, and files that are open during the backup are never a problem. Microsoft Volume
Shadow Copy Services (VSS) enables you to perform online backup of applications, which
traditionally is not possible. VSS is supported on the DS8000 storage server subsystem with
FlashCopy capabilities. VSS accomplishes this by facilitating communications between the
following three important entities:
򐂰 Requestors
An application that requests that a volume shadow copy be taken. These are applications,
such as backup or storage management, that request a Point-in-Time Copy of data or a
shadow copy.
򐂰 Writers
A component of an application that stores persistent information about one or more
volumes that participate in shadow copy synchronization. Writers are software that is

included in applications and services to help provide consistent shadow copies. Writers
serve two main purposes—by responding to signals provided by VSS to interface with
applications to prepare for shadow copy and providing information about the application
name, icons, files, and a strategy to restore the files. Writers prevent data inconsistencies.
򐂰 Providers
A component that creates and maintains the shadow copies. IBM VSS Provider is the
provider interface that interacts with the Microsoft Volume Shadow Copy Services and to
the Common Interface Model Agent (CIM Agent) on the master console.
Figure A-8 shows the Microsoft VSS architecture and how the software provider and
hardware provider interact through Volume Shadow Copy Services.
R
R ee qq uu ee ss ttoo rr
V oo l uu m ee S hh a d oo w
CC o p yy S S e rr v i cc e W r i t ee r ss
Apps
II//O
O
S
S oo fftt w
w aa rr ee H
H aa rr dd w
w aa rr ee
P
P rr oo vv iidd ee rr P
P rr oo vv iidd ee rr
Figure A-8 Microsoft VSS architecture
Microsoft Volume Shadow Copy Service function

Microsoft VSS accomplishes the fast Backup Process when a backup application initiates a
shadow copy backup. Microsoft VSS coordinates with the VSS-aware writers to briefly hold
writes on the databases, applications, or both. Microsoft VSS flushes the file system buffers
and asks a provider to initiate a FlashCopy of the data. Once the FlashCopy is logically
completed, Microsoft VSS allows writes to resume and notifies the requestor that the backup
has completed successfully. The volumes are mounted, hidden, and for read-only purposes,
to be used when rapid restore is necessary. Alternatively, the volumes can be mounted on a
different host and used for application testing or backup to tape.
The following list shows the Microsoft VSS FlashCopy process:
1. The requestor notifies Microsoft VSS to prepare for shadow copy creation.
2. Microsoft VSS notifies the application-specific writer to prepare its data for making a
shadow copy.
3. The writer prepares the data for that application by completing all open transactions,
flushing of cache, and writing in-memory data to disk.
4. When the data is prepared for shadow copy, the writer notifies the VSS, and it will relay
the message to the requestor to initiate the commit copy phase.
5. VSS temporarily quiesces application I/O write requests for a few seconds and the
hardware provider will perform the FlashCopy on the Storage Unit.
6. After the completion of FlashCopy, VSS releases the quiesce, and database writes will
resume.
7. VSS queries the writers to confirm that write I/Os were successfully held during Microsoft
Volume Shadow Copy.
Microsoft VSS with DS8000 FlashCopy

The following steps are performed by the Microsoft VSS in conjunction with FlashCopy when
a backup application initiates a request for backup on a DS8000 Storage Unit.
1. VSS retrieves a list of volumes from the DS8000 and selects appropriate target volumes
from the free pool (VSS_FREE).
2. VSS moves the target volumes to the reserved pool (VSS_RESERVED) and database
suspends on writes.
3. VSS issues a FlashCopy from the source volumes to the target volumes and database
resumes on writes after completion of FlashCopy.
4. VSS assigns the target volumes to the backup server’s Host Bus Adaptors (HBAs) where
Windows mounts the volumes on the backup server.
5. Requestor reads the data of the target volumes and copies it to tape.
6. Once the tape copy completes, Windows un-mounts the volumes and VSS unassigns
target volumes from the backup server’s HBAs.
7. VSS assigns target volumes back to the free pool (VSS_FREE).

You can refer to the FlashCopy diagram shown in Figure A-9.
R equestor
Backup
App W
W riters
riters
Apps
Volum
Volume
Shadow
C opy I/O
Service
D
DS8000
S8000 IB
IBM VSS Hardw are
Provider
Provider
Provider
VSS_R ESERVE VSS_FREE Production

Pool Pool Pool
FlashC opy
Target Source
W in2003 W in2003
Backup Production
Server Server
Figure A-9 Microsoft VSS with DS8000 FlashCopy
Additional information
You can refer to IBM TotalStorage DS Open Application Programming Interface Reference,
GC35-0493, for more technical information and implementation:
You can refer to the following Web sites for more information about Microsoft VSS:
http://www.microsoft.com/resources/documentation/WindowsServ/2003/all/techref/en-us/w2k3tr_
vss_how.asp
http://www.microsoft.com/windowsserversystem/storage/technologies/vss/default.mspx
Microsoft Virtual Disk Service (VDS)

Microsoft Virtual Disk Service is designed to meet the disk and storage subsystem
management needs for both small and midsize system configurations for direct attached or
SAN-based storage. This interface is available in Microsoft Windows 2003 Server and it is
designed to be able to scale up to enterprise configurations without compromising the
complex SAN functionality enabled by hardware vendors.
Virtua Disk Service overview

VDS is used as a single component within Microsoft Windows 2003 Server and serves as a
management tool. With Windows Server 2003, Microsoft introduced the Virtual Disk Service
(VDS). It is just a SAN storage management tool rather than a Business Continuity solution. It
allows Windows administrators to perform functions such as creating and deleting volumes
and management of volumes assigned to a server.
It unifies storage management and provides a single interface for managing block storage
virtualization. This interface is vendor and technology neutral, independent of the layer where
virtualization is done, operation system software, RAID storage hardware, or other storage
virtualization engines. Microsoft’s Virtual Disk Server enables the management of
heterogeneous storage systems, while leveraging both the client and provider’s APIs.
Microsoft VDS also supports automatic LUN configuration, which facilitates dynamic
reconfiguration by hardware in response to load or fault handling. Microsoft VDS is a software
interface for managing block storage virtualization. VDS is a vendor-specific DLL module that
integrates the vendor-specific solution into the overall VDS architecture. This interface
module makes it possible for the DS8000 storage server to interoperate with VDS-enabled
applications that utilize the VDS API in a Windows environment. VDS hides the complexities
associated with storage implementation from applications. VDS makes query and
configuration operations common across all devices it manages.
VDS is a set of Application Programming Interfaces (API) that use two sets of providers to
manage storage devices. The built-in VDS software providers enable you to manage disks
and volumes at the operating system level. VDS hardware providers supplied by the
hardware vendor enable you to manage hardware RAID arrays. Windows Server 2003
components that work with VDS include the Disk Management Microsoft Management
Console (MMC) snap-in, the DiskPart command-line tool, and the DiskRAID command-line
tool, which is available in the Windows 2003 deployment kit.
VDS utilizes a standardized interface through a graphic user interface (GUI), at the command
prompt, or through a command line interface (CLI). Figure A-10 shows the Microsoft VDS
architecture.
Command-Line Tools:
Disk Management Storage Management
•DiskPart
MMC Snap-in Applications
•DiskRAID
Virtual Disk Service
Software Providers: Hardware Providers

•Basic Disks
•Dynamic Disks
Other
Disk
Subsystem
HDDs LUNs
DS8000
Hardware
Microsoft Functionalitya
Functionality
Non-Microsoft Functionality
Figure A-10 Microsoft VDS architecture

The following minimum hardware is required for installing Microsoft Virtual Disk Service on a
Windows Server 2003 operating system:
򐂰 For Virtual Disk Services: DS8000 Storage Unit
򐂰 Common Information Model (CIM) agent
Virtual Disk Service product components

Microsoft Virtual Disk Service (VDS) controls the process of making storage accessible to
systems that need it. It is transparent to applications (or users) how the data is stored,
whether on a single physical disk or spanned across several disks (a logical unit) in terms of
data protection and performance. VDS can present a physical disk or a logical disk to a
server. VDS can:
򐂰 Create/delete logical units, assign number IDs (or LUNs).
򐂰 Unmask LUNs to server.
򐂰 Create partitions and volumes.
򐂰 Format the file system.
– Basic disks. VDS is used to partition each physical disk and to create the volumes that
can be mapped to drive letters for use. These volumes are known as simple volumes
and do not span multiple disks. Basic disks are the legacy disks and they do not offer
the same performance and data protection that dynamic disks offer.
– Dynamic disks. VDS can be employed to create dynamic disks, which can consist of
either simple volumes or multi-partition volumes. Multi-partition volumes physically
span more than a single disk but are logically considered as a single volume. Dynamic
disks can be spanned, striped (RAID 0), mirrored (RAID 1), or striped with parity (RAID
5). VDS can be used to expand dynamic disks to make more space available to a
volume.
The DS8000 interacts with the IBM VDS hardware provider to Microsoft VDS. The
implementation is based on the DS CIM Agent and Microsoft VDS, using CIM technology to
query storage subsystem information and manage LUNs. Microsoft Virtual Disk Service
together with Microsoft Virtual Shadow Copy Service forms a unified heterogeneous storage
systems solution that provides:
򐂰 Managing block storage virtualization
򐂰 Discovery of new storage
򐂰 Boot from SAN
򐂰 Shadow copy creation that relates to the storage subsystem’s FlashCopy capability
򐂰 Creation of consistent backups of open files and applications
򐂰 Creation of shadow copies for shared folders
򐂰 For backup, testing, and data mining purposes
IBM uses QLogic SANsurfer VDS Manager to interact with Microsoft Virtual Disk Service.
Figure A-11 shows the QLogic SANsurfer VDS Manager startup screen.
Figure A-11 QLogic SANsurfer VDS Manager startup
Microsoft Virtual Disk Server by itself as a single component does not provide you with a
disaster recovery solution. VDS is primarily designed as a SAN management tool that allows
you to perform disk management functions. VDS provides you with an effective tool with
which you can manage multi-vendor storage systems by a single interface.

With Microsoft VDS and VSS, the most distinct advantages coupled with IBM TotalStorage
products are booting from SAN and Shadow Copy (Microsoft Volume Shadow Copy Service)
creation functionality. In a disaster recovery solution scenario, if you have DS8000 storage
subsystems serving a Windows 2003 environment, you can integrate them with Microsoft
VSS and VDS. This means that you can effortlessly manage your overall SAN environment,
creating FlashCopies and offloading backup to backup servers without shutting down your
production application, and creating and assigning logical units and managing the SAN
storage environment. Figure A-12 shows the overall Microsoft VDS and VSS architecture.
R e q u e sto r
V o lu m e S h a d o w
C o p y S e rv ice W riters
rite rs
Apps
V
VSSSS V
VDDSS I/O
S
Sooftw
ftw aare
re H
H aard
rdww aare
re H
H aard
rdww aare
re
P
Pro
rovvid
ideerr P
PPro
rovid
videerr Pro vvid
ideerr
C om m and
L in e
In te rfa ce
V irtu a l S to ra g e
D isk
is k M gt
S e rv ice
rvice App
D isk M g t
Figure A-12 Microsoft VDS and VSS architecture
Figure A-13 shows the Microsoft VDS and VSS architecture combined with FlashCopy.
Command
Command
Requestor Line
Line
Backup Interface
Interface
App Writers
Writers
Virtual Storage
Storage
Disk Mgt
Apps Service App
App
Volume
Shadow
Copy I/O Disk
Disk Mgt
Mgt
Service
DS8000
DS8000 IBM VSS Hardware IBM VDS
Provider Provider
Provider
VSS_RESERVE VSS_FREE Production

Pool Pool Pool
FlashCopy
Target Source
Win2003 Win2003
Backup Production
Server Server
Figure A-13 Microsoft VDS and VSS with DS8000 storage subsystem
Additional information
You can refer to IBM TotalStorage DS Open Application Programming Interface Reference,
GC35-0493, for more technical information and implementation.
You can refer to the following Web site for more information about the Microsoft Virtual Disk
Service:
http://www.microsoft.com/windowsserversystem/storage/technologies/shadowcopy/stormgtusingvd
svss.mspx
SUN Solaris and Copy Services

In the following section we describe the actions that should be taken to perform Copy
Services functions and mount a target volume on a SUN Solaris server. Making a Copy
Services target volume available to the same server or to another server is possible. You can
use the DS CLI for automation and create scripts to automate your procedures and to prepare
your target mount point.
FlashCopy without a volume manager

In this section we describe how to access FlashCopy volumes under SUN Solaris without
volume manager software. Native commands are used to show how it is possible to access
the target volume after the Copy Services function has completed.

A FlashCopy is a point-in-time copy that immediately creates a copy of the current status of
the data. To ensure that the copied data is a consistent copy, which enables you to start up
the application from the FlashCopy target, the application has to be stopped. To ensure that
no other I/O will be issued, source volumes may be unmounted before the FlashCopy is
proceed.
The shell script to be run before the application that will use the FlashCopy target should
include the operations shown in Example A-2.
Example: A-2 Backup preparation process

#quiesce an application
insert the quiescing script here
#unmounting the source
umount /source
#start FlashCopy relationships
insert the FlashCopy ds cli script here using the option -wait on the mkflash command
mount /source
#and resume the application
insert the resuming script here
#check the target for consistency
fsck -y /dev/rdsk/cXtYdZsN
#if OK mount it
mount /dev/dsk/cXtYdZsN /target
The foregoing steps for a FlashCopy operation can be done by a script, as shown in
Example A-2. This gives an example of preparing a backup using FlashCopy. Once the
FlashCopy is created the target volumes can be mounted to a different host, which can take
the data from the target volumes and send them to the final backup destination by a backup
server.
Remote copy without a Volume Manager

A remote copy relation can be established at any time. Before the target volumes can be
used, all data must be copied from the source to the target volumes. When this is finished
there are the following possibilities to make use of the target volumes:
򐂰 Stop the application at the primary site and terminate or fail over the Remote Copy.
򐂰 If the applications should stay running at the primary site, consistency can only be
provided by Metro Mirror using a freeze prior to a failover or a terminate of the relation, or
by Global Mirror using a reverse of its FlashCopy at the remote site.
Once the Remote Mirror has been failed over or terminated, the remote host can mount the
target volumes and start the application.
Copy Services using VERITAS Volume Manager

In the following section we describe how to perform FlashCopy and Remote Mirror and Copy
on SUN Solaris systems with VERITAS Volume Manager (VxVM) support.
FlashCopy with VERITAS Volume Manager

In many cases, a user will make a copy of a volume so that the data can be used by a
different machine. In other cases, a user may want to make the copy available to the same
machine. VERITAS Volume Manager assigns each disk a unique global identifier. If the
volumes are on different machines, this does not present a problem. However, if they are on
the same machine, the user needs to take some precautions. For this reason, the steps that
need to be taken are different for the two cases.
FlashCopy to a different server

One common method for making a FlashCopy of a VxVM volume is to first unmount the
source volume, issue the FlashCopy, and import the new FlashCopy onto a second server. In
general, the steps for performing this process are as follows:
1. Unmount the source volume on Server A.
2. Unmount the target volume on Server B.
3. Invoke FlashCopy commands.
4. Mount the source volume on Server A.
5. Mount the target volume on Server B.
FlashCopy to the same server

The simplest way to make the copy available to the source machine is to export and offline
the source volumes. In Example A-3, volume vol1 is contained in Disk Group DG1. This Disk
Group consists of one device (c6t1d0s2). When that disk is taken offline, the FlashCopy
target becomes available to the source volume, and can be imported.
Example: A-3 Making a FlashCopy available by exporting the source volume

#halt I/O on the source by unmounting the volume
umount /vol1
#execute FlashCopy commands here
#deport the source volume group
vxdg deport DG1
#offline the source disk
vxdisk offline c6t1d0s2
#now only the target disk is online
#import the volume again
vxdg import DG1
#recover the copy
vxrecover -s Vol1
#re-mount the volume
mount /vol1
If you want to make both the source and target available to the machine at the same time, it is
necessary to change the private region of the disk, so that VERITAS Volume Manager allows
the target to be accessed as a different disk. Here we explain how to simultaneously mount
DS FlashCopy source and target volumes to the same host without exporting the source
volumes when using VERITAS Volume Manager. Check with VERITAS and IBM on the
supportability of this method before using it.
It is assumed that the sources are constantly mounted to the SUN host, the FlashCopy is
performed, and the goal is to mount the copy without unmounting the source or rebooting.
After the target volumes have been assigned, it is necessary to reboot the SUN server using
reboot -- -r or, if a reboot is not immediately possible, then issue drvconfig, disks, and
then devlinks. However, a reboot is recommended for guaranteed results.
It is also assumed that the appropriate actions in order to use the target volumes with the host
have already taken place (that is, devfsadm, vxdctl enable, and so on).
The following procedure refers to these names:

򐂰 mydg: The name of the diskgroup that is being created.
򐂰 da_name: The disk name shown under the DISK column in the vxdisk list output.

򐂰 last_daname: The disk is known to VxVM as shown under the DEVICE column in the
vxdisk list output. This is the output of the vxdisk list on SUN Solaris.
Use the following procedure to mount the targets to the same host:
1. Determine which disks have a copy of the disk group configuration in their private region.
The following command will list the log disk disks:
# vxdg list <disk group>
2. Determine the location of the private region (tag 15) on the disks (normally partition 3):
# prtvtoc /dev/rdsk/c#t#d#s2
Or use the following command to get the partition number for the private region:
# vxdisk list c#t#d#s2 | grep priv
3. Dump the private region:
# /usr/lib/vxvm/diag.d/vxprivutil dumpconfig /dev/rdsk/c#t#d#s3 > dg.dump
4. Create a script to initialize the disk group:
# cat dg.dump | vxprint -D - -d -F "vxdg -g <mydg> adddisk %name=%last_da_name" > dg.sh
5. Edit the file dg.sh and change the first line to:
# vxdg init <mydg> <daname>=<last_daname>
6. Make the file dg.sh executable:
# chmod 755 dh.sh
7. Create a file that can be used to rebuild the VM config:
# cat dg.dump | vxprint -D - -hvpsm > dg.maker
8. Initialize the disk group by executing dg.sh:
# ./dh.sh
9. If this results in the error Disk is already in use by another system, then the private
region on each disk that is to be added to the disk group will need to be initialized. This
can be done with the following command:
# vxdisksetup -i <da_name>
10. Rebuild the VM configuration:
# vxmake -g <mydg> -d dg.maker
11. Start the volumes:
# vxvol -g <mydg> start <volume>
Remote Mirror and Copy with VERITAS Volume Manager

In the previous section we described how to perform a FlashCopy and mount the source and
target file system on the same server. Here we describe the steps necessary to mount a
Remote Mirror and Copy secondary volume onto a server that does not have sight of the
primary volume. It assumes that the Remote Mirror and Copy copy pair has been terminated
prior to carrying out the procedure.
After the secondary volumes have been assigned, it is necessary to reboot the SUN server
using reboot -- -r or, if a reboot is not immediately possible, then issue drvconfig, disks
and then devlinks. However, a reboot is recommended for guaranteed results.
Use the following procedure to mount the secondary volumes to another host:
1. Scan devices in the operating system device tree:
#vxdisk scandisks
2. List all known disk groups on the system:
#vxdisk -o alldgs list
3. Import the Remote Mirror and Copy disk group information:
#vxdg -C import <disk_group_name>
4. Check the status of volumes in all disk groups:
#vxprint -Ath
5. Bring the disk group online:
#vxvol -g <disk_group_name> startall
or
#vxrecover -g <disk_group_name> -sb
6. Perform a consistency check on the file systems in the disk group:
#fsck -V vxfs /dev/vx/dsk/<disk_group_name>/<volume_name>
7. Mount the file system for use:
#mount -V vxfs /dev/vx/dsk/<disk_group_name>/<volume_name> /<mount_point>
Once you have finished with the Remote Mirror and Copy secondary volume, it is
recommended that you perform the following tasks:
1. Unmount the file systems in the disk group:
#umount /<mount_point>
2. Take the volumes in the disk group offline:
#vxvol -g <disk_group_name> stopall
3. Export disk group information from the system:
#vxdg deport <disk_group_name>
Tip: If you FlashCopy or Remote Mirror and Copy only one half of a RAID 1 mirror, it will be
necessary to force the import of the disk group because not all of the disks are available.
Therefore, it is necessary to issue the following command:
vxdg -f import <disk_group>
However, be aware that this may cause disk group inconsistencies.
HP-UX and Copy Services

The following section describes how it is possible to access a source and target Copy
Services volume on the same HP server.
HP-UX and FlashCopy

The following procedure must be followed to permit access to the FlashCopy source and
destination simultaneously on an HP-UX host. It could be used to make an additional copy of
a development database for testing or to permit concurrent development, to create a
database copy for data mining that will be accessed from the same server as the OLTP data,
or to create a Point-in-Time Copy of a database for archiving to tape from the same server.
This procedure must be repeated each time you perform a FlashCopy and want to use the
target physical volume on the same host where the FlashCopy source volumes are present in
the Logical Volume Manager configuration.

Target preparation
In order to prepare the target system, carry out the following steps:
1. Vary off the source volume groups:
#vgchange -a n /dev/<source_vg_name>
2. If you did not use the default Logical Volume Names (lvolnn) when they were created,
create a map file from your source volume group using the vgexport command:
#vgexport -m <map file name> -p /dev/<source_vg_name>
Tip: This map file needs to be ftp’ed to the target host.
3. If the target volume group exists, remove it using the vgexport command. The target
volumes cannot be members of a Volume Group when the vgimport command is run:
#vgexport -m /dev/null /dev/<target_vg_name>
4. Shut down or quiesce any applications that are accessing the FlashCopy source.
FlashCopy execution
To execute the procedure, you must carry out the following steps:
1. Unmount all file systems in the source volume group.
2. Perform the FlashCopy using the option -wait
3. Mount all the file systems in the source volume group.
4. When the FlashCopy is finished, change the Volume Group ID on each DS Volume in the
FlashCopy target. The volume ID for each volume in the FlashCopy target volume group
must be modified on the same command line. Failure to do this will result in a mismatch of
Volume Group IDs within the Volume Group. The only way to resolve this issue is to
perform the FlashCopy again and reassign the Volume Group IDs using the same
command line:
vgchgid -f </dev/rdsk/c#t#d#_1>...</dev/rdsk/c#t#d#_n>
Note: This step is not needed if another host is used to access the target devices.
5. Create the Volume Group for the FlashCopy target:

#mkdir /dev/<target_vg_name>
#mknod /dev/<target_vg_name>/group c <lvm_major_no> <next_available_minor_no>
Use the lsdev -C lvm command to determine what the major device number should be for
Logical Volume Manager objects. To determine the next available minor number, examine
the minor number of the group file in each volume group directory using the ls -l
command.
6. Import the FlashCopy target volumes into the newly created volume group using the
vgimport command:
#vgimport -m <map file name> -v /dev/<target_vg_name>
</dev/dsk/c#t#d#_1>...</dev/dsk/c#t#d#_n>
7. Activate the new volume group:
#vgchange -a y /dev/<target_vg_name>
8. Perform a full file system check on the logical volumes in the target volume group. This is
necessary in order to apply any changes in the JFS intent log to the file system and mark
the file system as clean.
#fsck -F vxfs -o full -y /dev/<target_vg_name>/<logical volume name>
9. If the logical volume contains a VxFS file system, mount the target logical volumes on the
server:
#mount -F vxfs /dev/<target_vg_name>/<logical volume name><mount point>
Once access to the FlashCopy target volume is no longer required, unmount the file systems
and vary off the volume group:
#vgchange -a n /dev/<target_vg_name>
If no changes are made to the source volume group prior to the subsequent FlashCopy, then
all that is needed is to vary on the volume group and perform a full file system consistency
check, as shown in steps 7 to 9.
HP-UX with Remote Mirror and Copy

When using Remote Mirror and Copy with HP-UX, it is similar to using FlashCopy, apart from
the fact that the volume group should be unique to the target server, so there should be no
need to perform the vgchgid command to change the physical volume to volume group
association. Here is the procedure to bring secondary volumes online to Remote Mirror and
Copy target HP-UX hosts:
1. Allow the copy pair volumes to go into duplex state using the catch-up operation or by
leaving the volumes to become synchronized.
2. Quiesce the source HP-UX application to cease any updates to the primary volumes.
3. Terminate the Remote Mirror and Copy pair relationship.
4. Rescan for hardware configuration changes using the ioscan -fnC disk command. Check
that the disks are CLAIMED using ioscan -funC disk. The reason for doing this is that the
volume group may have been extended to include more physical volumes.
5. Create the Volume Group for the Remote Mirror and Copy secondary. Use the lsdev -C
lvm command to determine what the major device number should be for Logical Volume
Manager objects. To determine the next available minor number, examine the minor
number of the group file in each volume group directory using the ls -l command.
6. Import the Remote Mirror and Copy secondary volumes into the newly created volume
group using the vgimport command.
7. Activate the new volume group.
8. Perform a full file system check on the logical volumes in the target volume group. This is
necessary in order to apply any changes in the JFS intent log to the file system and mark
the file system as clean.
9. If the logical volume contains a VxFS file system, mount the target logical volumes on the
server.
If changes are made to the source volume group, they should be reflected in the /etc/lvmtab
of the target server. Therefore, it is recommended that periodic updates be made to make the
lvmtab on both source and target machines consistent. As with the AIX importvg, there are
two alternatives:
򐂰 Using the “Permit read access from target” option:
a. If you are using Global Copy, issue go-to-sync to allow the volumes to go to duplex
state.
b. Once the volumes are in the duplex state, suspend the primary volume so that no
updates are reflected on the secondary volumes.
c. Export the source volume group information into a map file.

d. Export the old volume group definitions from the target host.
e. Run an ioscan to identify any new volumes that have been assigned to the target hosts
due to expansion of the source volume group.
f. Import the target volume group definition using the map file generated from the source
host.
g. Reestablish the Remote Mirror and Copy relationship, only copying cylinders
out-of-sync.
򐂰 Using NOCOPY Remote Mirror and Copy establish:
a. Quiesce all write I/O to the primary volumes, and unmount the source file systems.
b. If you are using Global Copy, issue the go-to-sync command to all the secondary
volumes to catch up.
c. Once the volumes are in duplex state, terminate the Remote Mirror and Copy
relationship.
d. Export the source volume group information into a map file.
e. Export the old volume group definitions from the target host.
f. Run an ioscan to identify any new volumes that have been assigned to the target hosts
due to expansion of the source volume group.
g. Import the target volume group definition using the map file generated from the source
host.
h. Establish the pairs relationship with the NOCOPY option, so that only the primary and
secondary have a copy pair relationship without updates.
i. Immediately suspend the primary volumes.
j. Mount the file systems and start the application at the source.
k. Some time later, reestablish the pairs, only copying the out-of-sync cylinders.
VMware ESX server and Copy Services

With the number of different guest operating systems supported by VMware, it is possible to
have a large number of scenarios where the DS8000 Advanced Copy Services can help
users meet their business requirements. Furthermore, each of these scenarios has a number
of possible permutations. The section is not intended to cover every possible use of Copy
Services with VMware; rather, it is intended to provide hints and tips that will be useful in
many different Copy Services scenarios.
When using Copy Services with the guest operating systems, the restrictions of the guest
operating system still apply. For example, there are some restrictions when using Copy
Services with Microsoft Windows dynamic disks, as discussed in “Windows and Remote
Mirror and Copy” on page 598. These restrictions still apply when the guest operating system
is Windows on VMware. In some cases, using Copy Services in a VMware environment may
impose additional restrictions.
Before using these techniques, check with IBM and the IBM System Storage DS8000 Host
Systems Attachment Guide, SC26-7917, for the latest information about the support available
for these solutions.
VMware ESX server and FlashCopy
FlashCopy can be used with either the VMware Console operating system or with guest
operating systems. In general, though, FlashCopy is used when a large amount of data needs
to be copied in a very short time. The VMware Console operating system is unlikely to have
huge files (the virtual machine files tend to be quite small), so FlashCopy is not commonly
used to copy virtual machine files. Instead, we focus on using FlashCopy with VMware guest
operating systems. This focus includes the use of virtual disk files stored on VMFS disks.
Recall that VMware guest operating systems can use DS8000 volumes in one of three ways:
򐂰 Virtual disk
– The volume is formatted with the VMFS file system.
– .vmdk files are created and presented to the guest operating system as a virtual disk.
– Maximizes virtualization capabilities, including a redo log.
򐂰 Raw physical disk
– The volume can be directly accessed as a System LUN in physical compatibility mode.
– In this mode, I/Os are passed through the virtualization layer to the DS8000 with
minimal modifications.
– Maximizes performance.
򐂰 Raw device mapping (RDM)
– The volume can be accessed as a System LUN in virtual compatibility mode.
– This mode combines the advantages of virtual disks and raw physical disks, allowing
relatively fast performance while maintaining the virtualization features of virtual disks,
such as a redo log.
Tip: When using virtual disks or raw device mappings with FlashCopy, take care which
type of Disk Mode you choose. Persistent and Nonpersistent are the easiest to manage. If
you want to use Undoable or Append, you will have to carefully manage the metadata and
be sure to copy it to the target virtual machine at the same time that you take the
FlashCopy.
Regardless of the type of virtual disk used, FlashCopy works in much the same way. This
section illustrates a method of using FlashCopy with each of these virtual disk types.
Target virtual machine configuration

Perhaps the easiest way to use FlashCopy with virtual machines is to create an equivalent
configuration on the target virtual machine before invoking FlashCopy. That is, if the source
virtual machine contains a 10 GB virtual disk on a 20 GB LUN, then you should create a 20
GB LUN to be used as the FlashCopy target, create a 10 GB virtual disk on it, and assign the
virtual disk to the target virtual machine. After the target is pre-configured, then you are ready
to use FlashCopy.
Restriction: If you do not shut down the target virtual machine first, your FlashCopied data
may not be accessible by the target virtual machine, and you may corrupt the FlashCopy
target volumes.
If you do not want to create the equivalent configuration prior to using FlashCopy, you can
instead take the following steps to get the target virtual machine to use the FlashCopy target
volumes.

Figure A-14 shows a virtual machine with three different DS8000 disks. SCSI 0:1 is a virtual
disk on a VMFS volume on one DS8000 LUN. SCSI 0:2 is a raw physical disk (System LUN in
physical compatibility mode), while SCSI 0:3 is a raw device mapping (System LUN in virtual
compatibility mode).
Figure A-14 Virtual machine source disks
Next, we make a FlashCopy of all three volumes, and then add all three volumes to the target
virtual machine. In the target virtual machine properties, choose the Hardware tab and then
Add Device. After clicking Add Device, choose Hard Disk. You will then have the choice of
Blank, Existing, or System LUN (as seen in Figure A-15).
Figure A-15 Virtual disk types
For virtual disks located on a VMFS volume, choose Existing. For the other two types of raw
disks, use System LUN/Disk. On the subsequent panels, you should choose the same
parameters that were chosen for the source machine.
Restriction: If your source and target virtual machines are located on the same physical
ESX Server machine, VMware will prevent you from using FlashCopy with virtual disk
located on a VMFS volume. System LUNs (both in physical compatibility mode and virtual
compatibility mode) do not have this limitation.
Now, the FlashCopy target volumes can be used as normal on the target machine.
FlashCopy within a virtual machine

In this first scenario, we will FlashCopy two LUNs of a virtual machine onto two other LUNs
assigned to that same virtual machine. Figure A-16 shows the two source LUNs, mounted as
the F: drive and the G: drive on the guest operating system. The F: drive is a raw System LUN
in physical compatibility mode, while the G: drive is a raw System LUN in virtual compatibility
mode. The two target LUNs are mounted as the H: and I: drives. H: is of the same type as F:,
while I: is of the same type as G:.
Figure A-16 FlashCopy within the same virtual machine

As with other FlashCopy operations with VMware, it is important to shut down the target
virtual machine (which, in this case, is the same as the source virtual machine). After that, we
perform the FlashCopy, and then re-start the virtual machine. Figure A-17 shows that the
FlashCopy operation was successful, and now the H: and I: drives have the same information
as the F: and G: drives.
Figure A-17 FlashCopy within the same virtual machine - After FlashCopy
FlashCopy between virtual machines
In this scenario, we will FlashCopy two LUNs from a virtual machine, and then use the target
volumes on a second virtual machine. Figure A-18 shows the two LUNs, mounted as the F:
drive and the G: drive on the guest operating system. As above, the F: drive is a raw System
LUN in physical compatibility mode, while the G: drive is a raw System LUN in virtual
compatibility mode. Both drives contain a directory called VM1.
Figure A-18 FlashCopy between virtual machines - Source virtual machine

A second virtual machine is also configured with two LUNs. For clarity, the target virtual
machine also has two LUNs mounted as the F: and G: drives, though any drive letters can be
used. Figure A-19 shows the two target LUNs. Like the source machine, the F: drive is a raw
System LUN in physical compatibility mode, while the G: drive is a raw System LUN in virtual
compatibility mode. However, unlike the source machine, both of these drives contain
directories named VM2, indicating that they were created on the second virtual machine.
Figure A-19 FlashCopy between virtual machines - Target virtual machine
Before issuing the FlashCopy, it is important to prepare both the source and target machines
to be copied. For the source machine, this typically means quiescing the applications,
unmounting the source volumes, and/or flushing memory buffers to disk. See the appropriate
sections for your operating systems for more information about this topic.
For the target machine, typically the target volumes must be unmounted. This prevents the
operating system from accidentally corrupting the target volumes with buffered writes, as well
as preventing users from accessing the target LUNs until the FlashCopy is logically complete.
With VMware, there is an additional restriction that the target virtual machine must be shut
down before issuing the FlashCopy. VMware also performs caching, in addition to any
caching the guest operating system might do.
After preparing both the source and target machines, issue the FlashCopy and then start your
target virtual machine. Figure A-20 shows that the second virtual machine’s files have been
replaced.
Figure A-20 FlashCopy between virtual machines - Target virtual machine after FlashCopy
Now the target LUNs are immediately usable.
FlashCopy between ESX Servers

Using FlashCopy with two separate ESX Server machines allows the additional possibility of
making copies of virtual disks located on VMFS volumes as well. Recall that VMware uses
unique volume identifiers for each VMFS volume. When you FlashCopy VMFS volumes and
attach them to the same ESX Server, the server does not allow the target volume to be used
because of the duplication volume identifier. However, this duplicate volume identifier is not a
problem when the FlashCopy target is used on a different disk.

Figure A-21 shows three volumes defined on the source virtual machine. The E: drive is a
virtual disk stored on a VMFS volume. As above, the F: drive is a raw System LUN in physical
compatibility mode, while the G: drive is a raw System LUN in virtual compatibility mode. All
three drives contain a directory named VM1 to show that they come from the source virtual
machine.
Figure A-21 FlashCopy between ESX Servers - Source virtual machine
On the target virtual machine on a second ESX Server machine, as seen in Figure A-22 on
page 625, the LUNs have been configured and assigned. Disk 1 is the target’s virtual disk on
a VMFS volume, while Disk 2 is a raw System LUN in physical compatibility mode, and Disk 3
is a raw System LUN in virtual compatibility mode. No Windows file systems have been
created, and no drive letters have been assigned to these volumes.
Again, in order to obtain a consistent copy of the source LUNs, you have to prepare both the
source and target virtual machines. This means quiescing the source volumes like you would
do when FlashCopying native Windows or Linux volumes. This also means that the target
virtual machine should be powered off. After these preparations, you are ready to issue the
FlashCopy command.
In Figure A-22, you can see that the target volumes have been assigned to the target virtual
machines, but no volumes have been created.
Figure A-22 FlashCopy between ESX Servers - Target virtual machine

After the FlashCopy, power on the target virtual machine. The target volumes are immediately
available. Once you mount the volumes or assign drive letters, you can see that the target
volumes contain the same data as the source volumes, as shown in Figure A-23.
Figure A-23 FlashCopy between ESX Servers - Target virtual machine after FlashCopy
On subsequent FlashCopies, you may not need to mount the volumes or assign drive letters
again. Unless you reconfigure, the target virtual machine will remember the previous mount
points.
VMware ESX server with Remote Mirror and Copy

It is possible to use Remote Mirror and Copy with all three types of disks. However, in most
environments, raw System LUNs in physical compatibility mode are preferred.
As with FlashCopy, using VMware with Remote Mirror and Copy contains all the advantages
and limitations of the guest operating system. See the individual guest operating system
sections for relevant information. Using VMware with Remote Mirror and Copy imposes some
additional restrictions.
One such limitation is the mkpprc -tgtread parameter is not supported. VMware cannot use
VMFS-formatted volumes or raw System LUNs in virtual mode without writing to the disk.
However, it may be possible to use raw System LUNs in physical compatibility mode. Check
with IBM on the supportability of this procedure.
At a high level, the steps for creating a Remote Mirror and Copy are as follows:
1. Shut down the guest operating system on the target ESX Server.
2. Establish remote mirror and copy from the source volumes to the target volumes.
Important: You should use the mkpprc -resetreserve flag when establishing the
Remote Mirror and Copy. Otherwise, you may receive this message on the target
machine:
Cannot create partition table for disk vmhbax:x:x because geometry info is invalid.
Please rescan
This flag removes hardware protection on the target device and should be used with
caution.
3. When the initial copy has completed and the volumes are now in Full Duplex mode,
suspend or remove the Remote Mirror and Copy relationship.
4. Issue the Rescan SAN command on the target ESX Server.
5. If not already assigned to the target virtual machine, assign the Remote Mirror and Copy
volumes to the target virtual machine. VMFS-formatted volumes should be assigned as
existing volumes, while raw volumes should be assigned as System LUNs of the same
type as the source.
6. Start the virtual machine and, if necessary, mount the target volumes.

B
Appendix B. Copy Services with System i5

This chapter provides descriptions of business continuity solutions with System i5™ and DS
systems, based on DS Copy Services.
The family of servers described in this chapter was introduced in 1988 as AS/400. In the
mid-nineties, the brand name was changed to iSeries. The name changed again recently to
System i or System i5. System i refers to entire product line, including older models, while
System i5 refers only to models based on the Power5 processor.
In this chapter we use the name System i5 because some of the solutions we discuss use
Boot from SAN and this function can only be realized on System i5. Note that other solutions
discussed here rely on Independent Auxiliary Storage Pools (IASPs), and these can be
implemented on older models as well.
Only in some instances when we want to point out a characteristic that apply to all models, do
we use the name System i.

Introduction
System i servers have been able to take advantage of Copy Services provided by external
disk storage (ESS) since 2001. At the beginning, clients were mainly implementing disaster
recovery solutions built on Remote Mirror and Copy (formerly PPRC). Over time, with the
evolving System i technologies and new external storage offerings, more and more System i
clients have addressed their business continuity issues by relying on external storage.
Today, System i5 clients have a variety of choices for external storage solutions and features.
With the DS6000 or DS8000 storage systems, they can use Metro Mirror or Global Mirror for
disaster recovery, they can replicate all disks or just application data, and they can use
FlashCopy to minimize downtime when taking backups.
This chapter presents and describes solutions to minimize downtime. First we review the
System i5 functions that are relevant to implementing external storage. An understanding of
those functions is critical to properly propose and manage external storage based solutions
and scenarios.
When discussing business continuity solutions for a System i5 audience, we use terms such
as continuous operations, high availability and continuous availability, which are usually
used within the System i community. We have included a formal definition in the note below.
Definitions:
򐂰 Continuous Operations (CO): Attribute of a system ability to continuously operate and
mask planned outages from end-users. It employs Non-disruptive hardware and
software changes, nondisruptive configuration, software coexistence.
򐂰 High Availability (HA): Attribute of a system ability to provide service during defined
periods, at acceptable or agreed upon levels, and mask unplanned outages from end
users. It employs fault tolerance, automated failure detection, recovery, bypass
reconfiguration, testing, and problem and change management
򐂰 Continuous Availability (CA): Attribute of a system ability to deliver non disruptive
service to the end user 7 days a week, 24 hours a day (there are no planned or
unplanned outages).
Continuous Operations + High Availability = Continuous Availability
System i5 functions and external storage

To better understand solutions using System i5 and DS storage systems, it is necessary to
have basic knowledge of System i5 functions and features that enable external storage
implementation and usage.
The following functions are discussed in this section:

򐂰 System i5 structure
򐂰 Single level storage
򐂰 Input output processors (IOPs)
򐂰 Clusters
򐂰 Independent Auxiliary Storage Pools (IASPs)
System i5 structure
System i5 is the newest generation of System i (formerly iSeries) servers. It is based on the
POWER5 processor. A System i5 can run one or multiple logical partitions, managed by an
hypervisor that resides in main memory.
A partition can host one of the following operating systems: i5/OS (formerly OS/400®), Linux,
or AIX.
Configure and manage partitions through a Hardware Management Console (HMC) that is
connected to the System i5 via an Ethernet connection.
System i5 partitions and HMC are depicted on Figure B-1.
Hardware Management i5/OS AIX LINUX i5/OS

Console
Service
Partition
SLIC SLIC
Firmware
Private
Desktop
Network
OR Firmware
and/or Perm | Temp
Public
Network
Rack mount
Figure B-1 System i5 partitions
In this chapter we discuss solutions with i5/OS partitions and DS systems. In the remaining of
this chapter, we refer to an i5/OS partition in System i5, simply as partition.
Single-level storage
For storage, the System i5 with i5/OS uses the same architectural component already used
by iSeries and AS/400, that is a single-level storage. This means that the System i5 sees all
disk space and the main memory as one storage area, and uses the same set of virtual
addresses to cover both main memory and disk space.
Appendix B. Copy Services with System i5 631

Paging in this virtual address space is performed in 4-KB pages. Single-level storage is
graphically depicted in Figure B-2.
I5/OS Partition Single-Level Storage
Main Memory
Figure B-2 Single level storage
When the application performs an input output (IO) operation, the portion of the program that
contains read or write instructions is first brought into main memory where the instructions are
then executing.
With the read request, the virtual addresses of the needed record are resolved and for each
needed page, storage management first looks to see if it is in the main memory. If the page is
there, it is used for resolving the read request. But if the corresponding page is not in the main
memory, it must be retrieved from disk (page fault). When a page is retrieved, it replaces a
page that was recently not used; the replaced page is swapped to disk.
Similarly, writing a new record or updating an existing record is done in main memory, and
the affected pages are marked as changed. A changed page remains in main memory until it
is swapped to disk as a result of a page fault. Pages are also written to disk when a file is
closed or when write to disk is forced by a user through commands and parameters. Also,
database journals are written to the disk.
A subject in System i5 is anything that exists and occupies space in storage and on which
operations can be performed. For example, a library, a database file, a user profile, a
program, are objects in System i5.
Input Output Processors

Input Output Processor (IOP) is a special card in the System i5 to which input output adapters
for disk, tape, LAN communication, and other similar devices are attached. The function of
IOP is to provide a certain degree of managing IO operations, and consequently off load
some of the I/O-related work from the System i5 central processor. Fibre Channel (FC)
adapters are used to connect external storage, attach to IOPs, and I/O operations to and from
external storage are managed by IOPs.
Some of the functions performed by an IOP with attached FC adapter include translating a
request for a block of data so that the adapter can understand it and informing the adapter of
the location of a data block in main memory.
The IOP concept is shown on Figure B-3.
System i5 Partition
Main memory
RIO
IOP IOP
PCI-X PCI-X
FC FC
ad ad
SAN
SAN
DS - External Storage
Figure B-3 Input Output Processors (IOPs)
Clusters
Many of System i5 continuous availability solutions which use external storage are based on
clusters and independent disk pools (independent auxiliary storage pools).
Clusters provide continuous availability in several ways: they use techniques like switched
disks, replicating data with Cross Site Mirroring, or replicating data with external disks while
using the copy services functions they provide. Continuous availability solutions from ISVs
also use System i5 clusters.
A System i5 cluster is a group of one or more systems or logical partitions that work together
as a single system.

The basic concepts related to cluster include cluster nodes, cluster resource group, and
domains:
򐂰 A cluster node is either a system or logical partition that is a member of the cluster. When
you create a cluster, you specify the systems or logical partitions that you want to include
in the cluster as nodes.
򐂰 The primary node is the cluster node that is the primary point of access for the resilient
cluster resource.
򐂰 A backup node is a cluster node that will take over the role of primary access if the present
primary node fails or a manual switchover is initiated.
򐂰 A cluster resource group (CRG) is an object in system i5 that represents a set of cluster
resources that are used to manage events that occur in a clustered environment. Different
types of CRG are used for represent different resources. For example, Application CRGs
are used to handle applications at different cluster events, data CRG are used for data
files, device CRG are used for devices. Within these types of CRGs there are two
common elements: a recovery domain and an exit program.
򐂰 A recovery domain defines the role of each node in the CRG. When you create a CRG in
a cluster, the CRG object is created on all nodes specified to be included in the recovery
domain. A recovery domain specifies the order of recovery for the nodes in cluster.
򐂰 A device domain is a subset of nodes in a cluster that share device resources. Nodes that
are in a device domain can participate at a recovery of a group of devices.
Elements of a cluster are shown on Figure B-4.
Elements of a Cluster Cluster
Device Cluster Node

CRG CRG
Domain A A
CRG
C
CRG CRG CRG
B B C
Recovery Cluster
Domain Resource
Group
Cluster Resources
(e.g., IASP)
Figure B-4 Elements of System i5 Cluster
Once the cluster is set up, the necessary information is maintained on all cluster nodes
through the heartbeating which runs on IP connection among the nodes.
If a system outage or a site loss occurs, the functions that are provided on a system or
partition within a cluster can be accessed through other systems or partitions that have been
defined in the cluster. When maintenance is needed on production partition, another node in
a cluster can handle resources of production partition and continues production work. This
functionality is achieved through cluster events like failover, switchover, replication, and
rejoin.
Solutions that are based on copy services provided by external storage systems and
Independent Auxiliary Storage Pool (IASP) require that all involved systems are in a cluster.
For more information about System i5 clusters refer to the iSeries Information Center on the
following Web page:
http://publib.boulder.ibm.com/iseries/
Also refer to the IBM Redbook Clustering and IASPs for Higher Availability on the
IBM Eserver iSeries Server, SG24-5194.
Independent Auxiliary Storage Pools (IASPs)

System i has a rich storage management heritage. From the start, the System i platform
made managing storage simple through the use of disk pools. For most customers, this
meant a single pool of disks called the System Auxiliary Storage Pools (ASPs). Automatic
use of newly added disk units, RAID protection, and automatic data spreading, load
balancing, and performance management makes this single disk pool concept the right
choice.
However, for many years, customers have found needs for additional storage granularity,
including the need to sometimes isolate data into a separate disk pool. Since the first AS/400,
many customers have utilized User ASPs to isolate disk units for journaling data (logging),
saving data to disk, and isolating slower, denser, less costly disk units for archive needs. User
ASP provide the same automation and ease-of-use benefits as the System ASP, but provide
additional storage isolation when needed. With software level Version 5, IBM takes this
storage granularity option a huge step forward with the availability of Independent Auxiliary
Storage Pools (IASPs).
An IASP is a collection of disk units that can be brought online or taken off-line independent of
the rest of the storage on a system. An IASP can be switched between systems or logical
partitions. And unlike User ASPs, IASPs coupled with i5/OS Clusters allow granular
replication based on the IASP, using storage options such as i5/OS Cross Site Mirroring or
IBM Storage Systems Copy Services.
IASPs represent a strategic direction for System i Storage Management. IASPs help keep the
System i promise of simplified IT management, while joining together the need for more
storage granularity, flexibility, and storage options
A system or partition has always a system ASP where vital system data reside. It can also
have user ASPs and IASPs. In partition with IASPs, we use the term sysbas. Sysbas refers to
system ASP plus user ASPs. When talking about solutions in such partition we usually
distinguish between IASPs and sysbas.

ASPs and IASPs in System i5 are shown on Figure B-5.
Auxiliary Storage Pools

(Disk Pools)
System ASP User ASPs

(Disk pool 1) (Disk Pools 2-255)
Traditional User ASPs

(Basic Pools 2-32)
Sysbas Independent ASPs - IASPs

( Independent Pools 33-255)
Figure B-5 IASPs in System i5
IASPs can be on internal disks in System i5 (can contain internal disks), or they can be on
external storage (contain volumes from ESS, DS6000, or DS8000).
When an IASP resides on external storage it can be replicated by DS6000 or DS8000 Copy
Services. The copy of an IASP can then be varied on to another System i or partition in a
cluster, making applications and data in that IASP available to the other system.
Continuous availability solutions for IASPs residing on DS6000 or DS8000 external storage,
are based on Metro Mirror or Global Mirror of IASP volumes. Solutions for minimizing
downtime for backups are based on FlashCopy of IASP volumes.
Metro Mirror for an IASP

This section describes Metro Mirror operations with an Independent Auxiliary Storage Pool
(IASP). In System i5 environment Metro Mirror is typically used to replicate data on distance
up to 50 km. For longer distances between local and remote site, we recommend to use
Global Mirror.
For more information about Metro Mirror refer to Part 4, “Metro Mirror” on page 107. For more
information about IASPs refer to “Independent Auxiliary Storage Pools (IASPs)” on page 635.
When describing functions of solutions with DS Copy services, we use terms planned outage,
unplanned outage and switchover. They are formally defined below.
Note: During a planned outage (also called a scheduled outage), you deliberately make
your system unavailable to users. You might use a scheduled outage to run batch work,
back up your server, or apply fixes.
An unplanned outage (also called an unscheduled outage) is usually caused by a failure.

You can recover from some unplanned outages (such as disk failure, system failure, power
failure, program failure, or human error) by using backups, or with disaster reocvery
solution. An unplanned outage that causes a complete system loss, such as a tornado or
fire, requires you to use disaster recovery solution and have a detailed disaster recovery
plan in place in order to recover.
A switchover is a manual process initiated by the user in order to move server functions
during a planned outage of the primary server to the backup server. This process involves
the quiescing of the application environment and system functions on the primary server
and the initiation of the application and system resource on the backup system. A
switchover can take anywhere from minutes to hours depending on the complexities of the
infrastructure and application environment.
Solution description
Our setup scenario for this solution consists of a local partition and a remote partition in
separate System i5 servers, but both partitions are grouped in a cluster.
Critical application data reside in an IASP in the local partition, and the IASP contains
volumes that reside on an external disk system. A Metro Mirror relationship is established
between the volumes in that IASP and volumes in another, remote disk system to which the
remote partition has access.
The Metro Mirror secondary volumes (in other words, the exact copy of the production IASP)
can be varied on to the recovery partition.
Note that it is only necessary that IASP on both partitions reside on external storage. sysbas
in each partition can be on internal disks or on external disk system. For more information
about sysbas refer to “Independent Auxiliary Storage Pools (IASPs)” on page 635.

Figure B-6 illustrates the setup for Metro Mirror of an IASP.
Connection for cluster

Production Recovery
Metro IASP
Mirror
IASP
Figure B-6 Metro mirror of IASP
This solution provides continuous availability in case of planned and unplanned outages:
򐂰 For each planned or unplanned outage, the Metro Mirror copy of the production IASP is
made available to the remote partition which continues to run production application from
the data in the mirrored IASP.
򐂰 When the planned outage is finished or the cause of failure of the unplanned outage has
been fixed, a reverse of the Metro Mirror direction (from secondary volumes to primary
volumes) is performed to transfer updated data in the remote IASP back to the local
partition. When the transfer is complete, the Metro Mirror relationship is established again
in the original direction (from the primary to the secondary volume) and production can
resume at the primary site.
This solution is implemented with the System i Copy Services Toolkit available for purchase
from the System i Client Technology Center in Rochester.
Note: The System i Copy Services Toolkit is a software product that gets installed on all
partitions in a FlashCopy or PPRC clustered environment.
Solution benefits
Metro Mirror of an IASP offers several benefits:
򐂰 Basically, recovery takes as long as is needed to vary on an IASP. Therefore recovery
time is significantly shorter than with solutions which use Boot from SAN, or external
loadsource, where IPL is needed for recovery. Recovery time with Metro Mirror of an IASP
can be compared to the recovery time for high availability solutions provided by ISVs.
Note that Boot from SAN is an IBM implementation of an external boot disk in System i. In
this implementation a boot disk is connected through a special input output processor
(IOP) and an FC adapter in System i5. For more information about Boot from SAN refer to
the IBM Redbook IBM System Storage DS6000 Series: Architecture and Implementation,
SG24-6781. External loadsource (boot disk) is also implemented by competitor’s external
storage solutions, though it doesn’t use Boot from SAN IOP.
򐂰 Due to synchronous copying of writes, Metro Mirror has always some impact on
production performance. With this solution we only use Metro Mirror of data in an IASP.
Therefore impact on performance is relatively small. On the other hand, solutions with
Boot from SAN or external loadsource replicate the entire partition disk space, including
temporary files, job queues, and so on and the performance degradation of the production
system can be more significant.
򐂰 This solution only has impact on write operations performances. It does not use any of the
System i5 CPU resources. This is different from some high availability implementations
provided by other vendors, where there is an impact to performances due to CPU usage
as part of their solution.
򐂰 With this solution we can run a non production workload, like test or development tasks in
the remote partition. This is again different from the implementation using Boot from SAN
where the recovery parititon has to be in stand-by, typically without any workload.
򐂰 The System i Copy Services Toolkit which is used for this solution provides full automation
of steps performed at planned or unplanned outages. All the needed steps for failover and
failback are initiated by one command in a partition of System i5.
򐂰 Once this solution is established it requires very little maintenance and administration.
Therefore some customers may prefer it to other high availability solutions available in
System i5.
Planning and requirements

The following points should be taken into account when planning for Metro Mirror of an IASP:
򐂰 The customer application must be implemented with the IASP before installing the System
i Copy Services Toolkit. The customer can engage IBM services for this. In any case, the
application setup in an IASP must be well panned before establishing the Metro Mirror.
򐂰 For successful implementation it is essential to contact the Rochester Toolkit team well in
advance. Alternatively, you can contact Advanced Technical Support.
Note: For information about the Toolkit contact the Client Technology Center, via the
following Internet page:
http://www-03.ibm.com/servers/eserver/services/iseriesservices.html
򐂰 Solutions that use the System i Copy Services Toolkit cannot be sold to the customer
without prior approval by the Client Technology Center.
򐂰 The solution requires a System i and a DS6000, DS8000, or ESS at both local and remote
sites. It also requires appropriate FC connections between the DS systems for Metro
Mirror and connections between Systems i5 for cluster communication via TCP/IP.
򐂰 Each IASP has its own Fibre Channel attachment cards. This is valid for both System i5
and earlier System i models like 8xx and 270.
򐂰 A Metro Mirror license on both local and remote disk systems is required.
򐂰 i5/OS software for clustering (5722-SS1 option 41) must be installed in both production
and recovery partition. This prerequisite is needed on each System i5 and earlier System i
models like 8xx and 270.

򐂰 The following are software prerequisites on both System i5 and earlier models of System i
like the 8xx and 270:
– i5/OS licensed product JDK™ 1.4.2 (5722-JV1 Option 6) and the latest JAVA group
PTF
– i5/OS licensed product Crypto Access Provider 128-bit (5722-AC3)
– i5/OS software Portable Application Solutions Environment (PACE) (5722-SS1 Option
33)
– i5/OS licensed product IBM HTTP Server (5722-DG1)
– i5/OS licensed product Digital Certificate Manager (5722-SS1 Option 34)
򐂰 The i5/OS licensed product Secure Shell, SSH (5733-SC1) is needed on System i5.
However, it is not needed on earlier System i models.
򐂰 Careful sizing of links for Metro Mirror is needed. Follow these steps to size Metro Mirror
links (for detailed information, refer to the IBM Redbook iSeries and IBM TotalStorage: A
Guide to implementing external disk on IBM eServer i5, SG24-7120):
– Collect system i5 performance data. It is best to collect it over one week, and if
needed, during heavy workload such as when running end-of-month jobs.
– If the customer has already established IASP, observe in performance reports the
number of writes that go to the IASP. If the IASP is not set up yet, observe the number
of writes that go to the database.
• Multiply the writes to IASP by the reported transfer size to get the write rate
(MB/sec) for the entire period of collecting performance data.
• Look for the highest reported write rate. Size the number of Metro Mirror links so
that their bandwidth can accommodate the highest write rate.
The Client Technology Center can perform an in-depth analysis of bandwidth
requirements by using testing workload PEX if more accurate sizing is needed.
Considerations
Following points should be considered with this solution:
򐂰 An application in System i5 typically consists of different objects, like programs, data files,
data areas, spool files user profiles and others. After the application is set up in an IASP,
the majority of application objects reside in the IASP. However, some objects may reside
in sysbas. For example, data files that belong to the application go to the IASP while the
programs and job queues may stay in sysbas.
It is necessary to maintain application objects in sysbas on both production and recovery
partitions in sync, to allow the application to run in the recovery parititon after Metro Mirror
copy of the production IASP is made available to it.
As experienced with many applications, maintenance of sysbas is rather simple and
straightforward: it is achieved by i5/OS change control, or by installing new application
versions on both partitions at the same time. However, some customers may want to use
other ways of maintaining sysbas, like regularly saving the objects in the production
partition and restoring them in the recovery partition, or using a limited version of an ISV
solution.
For more information about i5/OS change control refer to the iSeries Information Center
on the following Web page and look for Management Central:
http://publib.boulder.ibm.com/infocenter/iseries/v5r4/index.jsp
i5/OS release V5R4 brings significant improvement in replicating i5/OS user profiles that
reside in sysbas. With this release it is possible to define a new type of cluster domain,
administrative domain, which enables that changes in user profiles are automatically
propagated from the local sysbas to remote sysbas. For more information about cluster
domains refer to “Clusters” on page 633.
򐂰 We strongly recommend journaling the application objects when using Metro Mirror for an
IASP, or even Metro Mirror of an entire disk space.
Important: When using Copy Services functions such as Metro Mirror, Global Mirror, or
FlashCopy for the replication of the load source unit or other i5/OS disk units within the
same DS6000/DS8000 or between two or more DS6000 and DS8000 systems, the source
volume and the target volume characteristics must be identical. The target and source
must be of matching capacities and matching protection types.
Additionally, once a volume is assigned to an i5/OS partition and added to that partition’s
configuration, its characteristics must not be changed. If there is a requirement to change
some characteristic of a configured volume, it must first be completely removed from the
i5/OS configuration. After the characteristic changes are made, for example, protection
type, capacity, and so on, by destroying and recreating the volume or by utilizing the DS
CLI, the volume can be reassigned to the i5/OS configuration. To simplify the
configuration, we recommend a symmetrical configuration between two IBM Storage
systems, creating the same volumes with the same volume ID that decides the LSS_ID.
Implementation and usage

This solution is implemented by services which come with the System i Copy Services
Toolkit. The following steps are performed by toolkit services to implement the solution:
1. Enable System i5 Hardware Management Console (HMC) to communicate with the
Toolkit.
2. Install needed software products on both production and recovery System i5.
3. Configure communication between Toolkit code and HMC.
4. Configure needed TCP/IP server.
5. Set up production and remote partitions in a cluster.
6. Create IASP on recovery parititon.
7. Install the toolkit programming code.
8. Create needed cluster resources.
9. Install and configure DS CLI on System i5, and create needed DS CLI profile, password
file and DS CLI scripts on System i5.
After the toolkit is installed and tested you can perform switchover and failback during
planned and unplanned outages by using toolkit commands. The following toolkit commands
are used:
򐂰 swpprc - You use this command with parameters *scheduled or *unscheduled depending
on whether a planned or unplanned outage is being handled. The swpprc command
initiates a sequence of steps in the Toolkit by which the required actions (switchover,
failback, and other actions) are achieved.
򐂰 You can also check whether the solution is in the correct status by using the System i5
chkpprc and dspessdta commands from the toolkit.

򐂰 Results of Toolkit actions at switch and when checking status, can be observed in the
Toolkit log by using command viewlog.
The listed commands are described later in this section.
When such a System i5 command is executed, the Toolkit triggers the different tasks by
communicating with the System i5 HMC, performing System i5 commands within the
partition, and communicating with external disks systems by using the DSCLI. This is
illustrated in Figure B-7.
iSeries Cluster /Device Domain/Data CRG
SSH HMC
HMC
TCP/IP SSH
TCP/IP
DS/ ESS
Toolkit Toolkit
FSP FSP
IASP IASP’
DS CLI DS CLI
(FIBRE)
(FIBRE)
IOA IOA
PPRC / FlashCopy Relationship
Figure B-7 Functioning of the System i Copy Services Toolkit
Below we describe the sequence of steps required for a switchover, for checking the Metro
Mirror relationship status, and for setting up the toolkit.
Note that in the examples shown below, the partition ITCHA2 is a local parititon, ITCHA3 is a
remote partition, and the IASP name is DS6000.
Checking the solution components

You may want to check the status of the following solution components:
򐂰 IASP
򐂰 Metro Mirror
򐂰 Toolkit
򐂰 DS CLI scripts
򐂰 Toolkit log
Checking IASP status
To observe the logical volumes in the IASP, use the System i5 graphical interface known as
iSeries Navigator, which is a part of the iSeries Access for Windows licensed product.
Instructions on how to install iSeries Access for Windows can be found in the iSeries
Information Center located at:
In iSeries Navigator make a connection to the System i5 that contains the IASP.
Note: Thew System i5 user ID and password are required to connect to the System i5.
After the connection is established, look for the system in the left-hand side panel of the
iSeries Navigator window and expand it. Then expand Configuration and Service →
Hardware → Disk Units → Disk Pools.
Note that for expanding disk units you will need a System i5 Service Tools user ID and
password.
Observe that IASPs are numbered 33 and higher, while traditional disk pools (user ASPs)
have numbers below 33. Click the relevant IASP to show its disks in the right-hand side
panel. This is shown in Figure B-8.
Figure B-8 Checking status of IASP
To check the status of the IASP, you can also use the i5/OS wrkcfgsts command.
After entering this command, press F4, which brings up a screen where you can insert
parameters. At *type insert *dev, at Configuration description, insert the name of IASP. Leave
parameter Output as *, as shown in Figure B-9 on page 644, and press Enter twice.

Attention: In the example shown in Figure B-9, the name of IASP is DS6000. Be aware
that DS6000 does not indicate a specific disk system here: it is only the name given to the
System i5 disk pool.
Work with Configuration Status (WRKCFGSTS)
Type choices, press Enter.
Type . . . . . . . . . . . . . . *dev *NWS, *NWI, *LIN, *CTL, *DEV

Configuration description . . . ds6000 Name, generic*, *ALL, *CMN...
Output . . . . . . . . . . . . . * *, *PRINT
Bottom
F3=Exit F4=Prompt F5=Refresh F10=Additional parameters F12=Cancel
F13=How to use this display F24=More keys
Figure B-9 wrkcfgsts
The next screen, as shown in Figure B-10. displays the name and status of the IASP.
Work with Configuration Status ITCHA2

06/25/06 11:51:29
Position to . . . . . Starting characters
Type options, press Enter.

1=Vary on 2=Vary off 5=Work with job 8=Work with description
9=Display mode status 13=Work with APPN status...
Opt Description Status -------------Job--------------

DS6000 AVAILABLE
Bottom
Parameters or command
===>
F3=Exit F4=Prompt F12=Cancel F23=More options F24=More keys
Figure B-10 Status of IASP
Checking the Toolkit setup

To observe the currently used Toolkit options, execute the dspessdta command on either the
production or remote partition of a System i5. To achieve this, establish a Telnet connection
using IBM Personal Communications, with the appropriate System i5 partition. For more
information about how to use IBM Personal Communications to Telnet to a System i5, refer to
the iSeries Information Center located at:
When the Telnet connection is established, specify the System i5 user ID and password to
sign in. Type dspessdta in the command line of the System i5 Telnet screen, as shown in
Figure B-11, and press Enter.
* MAIN OS/400 Main Menu

System: ITCHA1
Select one of the following:
1. User tasks
2. Office tasks
3. General system tasks
4. Files, libraries, and folders
5. Programming
6. Communications
7. Define or change the system
8. Problem handling
9. Display a menu
10. Information Assistant options
11. iSeries Access tasks
90. Sign off
Selection or command
===> dspessdta
F3=Exit F4=Prompt F9=Retrieve F12=Cancel F13=Information Assistant

F23=Set initial menu
Figure B-11 dspessdta

This brings up the screen shown in Figure B-12 where you insert the the IASP name and then
press Enter.
Change or Display ESS IASP Data
Type IASP name, press Enter.
Independent ASP Name .. . . . . DS6000 Name
F1=Help F3=Exit F12=Cancel
Figure B-12 IASP name
A new screen now displays the IASP name and other options or status information such as
the Metro Mirror direction, auto-start of cluster after link failure, and so on. This is shown in
Figure B-13. We recommend using this command and observing the options and status
before performing a switchover. After you have checked the values displayed for the different
fields, press F3 or F12 to exit this screen and will bring back the i5/OS command line.
Display ESS IASP Data
Press Enter to continue.
Independent ASP Name . . . . . : DS6000

Copy type . . . . . . . . . . : *BOTH
Request type . . . . . . . . . : 0
FlashCopy status . . . . . . . : *NONE
PPRC status . . . . . . . . . : *READY
PPRC direction . . . . . . . . : *NORMAL
Auto start cluster . . . . . : *YES
Enable FlashCopy scripts . . : *YES
Enable PPRC scripts . . . . . : *YES
Automatic PPRC Replicate . . : *YES
Multipath . . . . . . . . . . : *YES
Warm FlashCopy . . . . . . . . : *YES
Device CRG name . . . . . . . :
Wait time . . . . . . . . . . : 60
Message Queue . . . . . . . . : *SYSOPR
Library . . . . . . . . . . :
Bottom
Figure B-13 Display Toolkit options
Checking the Metro Mirror status
To check for a correct status of the DS storage systems prior to the switchover, use the
System i5 chkpprc command and specify the IASP name, as is shown in Figure B-14. We
recommend regularly performing the chkpprc command to be sure that DS systems are in a
correct status for switching to the remote site. You can execute this command in either the
local or recovery partition.
MAIN i5/OS Main Menu

System: ITCHA2
1. User tasks
2. Office tasks
5. Programming
6. Communications
8. Problem handling
9. Display a menu
90. Sign off
===> CHKPPRC ds6000

Figure B-14 chkpprc

During execution of this command, the Toolkit checks connection to the DS system and
displays online information at the bottom of the screen, as shown in Figure B-15.

System: ITCHA2
1. User tasks
2. Office tasks
5. Programming
6. Communications
8. Problem handling
9. Display a menu
90. Sign off
===> CHKPPRC ds6000

Checking lspprc DSCLI script for production node.
Figure B-15 Checking PPRC
Finally you are presented the message that PPRC was successfully checked, or you are
informed about a failed check. This is shown on Figure B-16 on page 649.
System: ITCHA2
1. User tasks
2. Office tasks
5. Programming
6. Communications
8. Problem handling
9. Display a menu
90. Sign off
===> CHKPPRC ds6000

A PPRC check for IASP CRG DS6000 completed successfully.
Figure B-16 chkpprc completed

You may also check results in the log after the chkpprc command has been executed. To
achieve this, enter the viewlog command at the System i5 command line. This is shown in
Figure B-17.

System: ITCHA2
1. User tasks
2. Office tasks
5. Programming
6. Communications
8. Problem handling
9. Display a menu
90. Sign off
===> viewlog

Figure B-17 viewlog
You are presented with the Toolkit log, where you can see the results of checking the Metro
Mirror status, as shown in Figure B-18.
Edit File: /tmp/qzrdiash.log

Record : 1721 of 1733 by 8 Column : 1 246 by 74
Control :
CMD ....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+
Fri Jun 23 08:01:16 2006 workWithHmcProfile: FYI, /QIBM/Qzrdiash5/hmc/DS600
"21020003/none/1,21030002/none/1,21050002/none/0,21010003/none/1,21030003/
Fri Jun 23 08:01:16 2006 doPPRCScript: Executing: QDSCLI/DSCLI SCRIPT('/QIB
Fri Jun 23 08:01:25 2006 checkThoseResults: Processing file /QIBM/Qzrdiash5
Fri Jun 23 08:01:25 2006 checkThoseResults: Strings Full Duplex -Target Ful
Fri Jun 23 08:01:25 2006 doPPRCScript: Results ok.

Fri Jun 23 08:01:25 2006 doPPRCScript: Executing: QDSCLI/DSCLI SCRIPT('/QIB
Fri Jun 23 08:01:33 2006 checkThoseResults: Processing file /QIBM/Qzrdiash5
Fri Jun 23 08:01:33 2006 checkThoseResults: Strings Target Full Duplex Metr
Fri Jun 23 08:01:33 2006 doPPRCScript: Results ok.

Fri Jun 23 08:01:33 2006 chkpprc: SUCCESSFUL, ready for the SWPPRC command.
************End of Data********************
F2=Save F3=Save/Exit F12=Exit F15=Services F16=Repeat find

F17=Repeat change F19=Left F20=Right
Figure B-18 Toolkit log
Checking DS CLI profile

To check the DS CLI profile in the System i5 partition, perform the following steps:
1. Enter the viewprof command. This brings up a screen where you can insert the IASP
name, as shown in Figure B-19. Press Enter.
View Profiles (VIEWPROF)

Independent ASP name . . . . . . ds6000 Name

System name . . . . . . . . . . *LOCAL Character value
Bottom
F3=Exit F4=Prompt F5=Refresh F12=Cancel F13=How to use this display
F24=More keys
Parameter IASPNAME required.
Figure B-19 Viewprof

2. Next you are presented with the screen showing profiles used in the partition. Look for the
profile you want to check and type 5 in the Opt field next to that profile name, as is shown
in Figure B-20.
Directory: /qibm/qzrdiash5/profiles/DS6000
Position to : Record : 1 of 4
New File :
2=Edit 4=Delete File 5=Display 6=Path Size 9=Recursive Delete
Opt Name Size Owner Changed Used

5 pprc_A.profile 8K QSECOFR 06/22/06 17:02 07/06/06 11:22
pprc_B.profile 8K QSECOFR 06/22/06 17:02 07/05/06 10:34
pprc_Asec.dat 8K QSECOFR 06/22/06 10:58 07/05/06 10:32
pprc_Bsec.dat 8K QSECOFR 06/22/06 10:58 07/05/06 10:34
Bottom
F3=Exit F5=Refresh F12=Cancel F16=Sort F17=Position to

F22=Display entire field
Figure B-20 DS CLI profiles on i5/OS
3. Press Enter to get the screen showing the selected DS CLI profile, as shown in
Figure B-21.
Browse : /qibm/qzrdiash5/profiles/DS6000/pprc_A.profile
Control :
....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....
************Beginning of data**************
#
# DS CLI Profile
#
#
# Management Console/Node IP Address(es)
# hmc1 and hmc2 are equivalent to -hmc1 and -hmc2 command options.
hmc1: 9.5.110.66
#hmc2: 127.0.0.1
pwfile: /qibm/qzrdiash5/profiles/DS6000/pprc_Asec.dat
#
# Default target Storage Image ID
F3=Exit F10=Display Hex F12=Exit F15=Services F16=Repeat find

F19=Left F20=Right
Figure B-21 Display DS CLI profile
Checking a DS CLI script
To check a DS CLI script in a System i5 partition, enter the viewscript command with the
IASP name as the parameter. You are presented with a screen listing the DS CLI scripts used
in this partition. Look for the script you want to examine and type 5 in the Opt field next to this
script, as is shown in Figure B-22. Press Enter.
Directory: /qibm/qzrdiash5/scripts/DS6000
Position to : Record : 1 of
New File :
2=Edit 4=Delete File 5=Display 6=Path Size 9=Recursive Delete
Opt Name Size Owner Changed Used

lspprc_A.script 8K QLPAR 06/22/06 16:23 07/05/06 10:32
lspprc_B.script 8K QLPAR 06/22/06 16:23 07/05/06 10:34
<overtask_A_1.script 8K QLPAR 06/22/06 16:23 07/05/06 10:34
<overtask_A_2.script 8K QLPAR 06/22/06 16:23 07/05/06 10:34
<overtask_B_1.script 8K QLPAR 06/22/06 16:23 06/22/06 16:24
<overtask_B_2.script 8K QLPAR 06/22/06 16:23 06/29/06 17:32
mkpprc_A.script 8K QLPAR 06/22/06 16:23 07/03/06 07:50
pausepprc_A.script 8K QLPAR 06/22/06 16:23 06/22/06 16:23
resumepprc_A.script 8K QLPAR 06/22/06 16:23 06/22/06 16:23
lspprcpath_A.script 8K QLPAR 06/22/06 16:23 06/22/06 16:23
5 pausepprc_B.script 8K QLPAR 06/22/06 16:23 06/22/06 16:23
resumepprc_B.script 8K QLPAR 06/22/06 16:23 06/22/06 16:23
lspprcpath_B.script 8K QLPAR 06/22/06 16:23 06/22/06 16:23
lspprc_A.result 8K QLPAR 07/05/06 10:32 07/05/06 10:32
More...
F3=Exit F5=Refresh F12=Cancel F16=Sort F17=Position to

F22=Display entire field
Figure B-22 Select DS CLI script on i5/OS

The selected script is displayed, as shown in Figure B-23.
Browse : /qibm/qzrdiash5/scripts/DS6000/pausepprc_B.script
Control :
....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....
************Beginning of data**************
# Generated: Thu Jun 22 16:23:10 2006
pausepprc -dev IBM.1750-6877801 -remotedev IBM.1750-13ADLWA 1200-1204:1200-1204
************End of Data********************
F3=Exit F10=Display Hex F12=Exit F15=Services F16=Repeat find

F19=Left F20=Right
1 records folded.
Figure B-23 DS CLI script on i5/OS
Switch to remote site at planned outages

To switch the production system to the remote site in planned outage situations, use the
swpprc command in the remote partition and press F4, as shown in Figure B-24.
MAIN OS/400 Main Menu

System: ITCHA1
1. User tasks
2. Office tasks
5. Programming
6. Communications
8. Problem handling
9. Display a menu
90. Sign off
===> swpprc

Figure B-24 swpprc
This brings up a screen where you can insert the IASP name and specify the parameter
*scheduled or *unscheduled. If you perform a switch to the production system for planned
outage specify *scheduled, if you switch to production for an unplanned outage, specify
*unscheduled.
Press Enter to execute the command. This is shown in Figure B-25.
Switch PPRC (SWPPRC)

Switch type . . . . . . . . . . *SCHEDULED *SCHEDULED, *UNSCHEDULED...
Bottom
F24=More keys
Figure B-25 Parameter scheduled
The swpprc toolkit command executes all the required actions to switch over to the IASP at
the recovery site, for both systems and external storage. The following actions take place
during the execution of swpprc *scheduled:
1. Run chkpprc command.
2. Vary off the IASP on local partition.
3. Perform Metro Mirror failover to secondary site.
For more information about Metro Mirror failover refer to 12.3, “Failover and Failback” on
page 115.
4. Perform Metro Mirror failback from secondary to primary site. For more information about
Metro Mirror failback refer to 12.3, “Failover and Failback” on page 115.
5. Vary on the IASP for remote partition.
6. By using System i5 logical partitioning functions, remove IOPs and adapters logically from
the former production system and modify the description of the local System i5 partition.

During its execution, the swpprc command displays messages about the currently executed
step at the bottom of the screen, as shown in Figure B-26.

Bottom
F24=More keys
Checking lspprc DSCLI script for production node.
Figure B-26 Messages about progress-2
Some other messages shown during execution of swpprc are as follows:

򐂰 Sent inquiry message to production system operator.
򐂰 Starting DS CLI Failover of ITCHA2 to ITCHA3.
򐂰 Starting DS CLI Replicate of ITCHA3 to ITCHA2.
򐂰 Adding IOAs to the backup HMC Profile.
򐂰 Removing IOAs from the production cluster node.
򐂰 Removing IOAs from the production HMC Profile.
After swpprc *scheduled is performed you can use the viewlog command to check if it was
succesfuly executed. An example is shown in Figure B-27.

Control :
CMD ....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+
Fri Jun 23 09:13:22 2006 workWithHmcProfile: FYI, issued command:
chsyscfg -r prof -m "ITCHA3" -i "name=ITCHA3,lpar_name=ITCHA3,io_slots=\\\
Fri Jun 23 09:14:23 2006 startDasdMgmtOperaton: First of 2 ASP records retu
Fri Jun 23 09:14:23 2006 startDasdMgmtOperaton: MultiPath has been reset fo
Fri Jun 23 09:14:23 2006 startDasdMgmtOperaton: Record 2 of 2 ASP records r
Fri Jun 23 09:15:40 2006 addRemoveHmcDevices: Removed device at bus 2, slot
Fri Jun 23 09:15:40 2006 addRemoveHmcDevices: Removed a total of 1 device(s
Fri Jun 23 09:15:41 2006 workWithHmcProfile: FYI, device on ITCHA2 bus 2 sl
Fri Jun 23 09:15:53 2006 swpprc: SUCCESSFUL.
************End of Data********************

Figure B-27 Log of SWPPRC
You can check the status of the IASP copy, which is now available at the remote partition.
Proceed as follows:
1. In the remote partition, enter the command wrkscfgsts cfgtype(*dev) cfgd(ds6000).
This will bring up a screen showing the IASP status. After successful switchover, the IASP
should have a status of available, as shown on Figure B-28.
Work with Configuration Status

System: ITCHA3
DS6000 AVAILABLE
Bottom
===>
F21=Select assistance level
Figure B-28 Metro Mirror copy if IASP after switchover
2. Since varying off the production IASP is done as a part of swpprc, the IASP is no longer
available to the local partition after the switchover. By using the command wrkcfgsts

cfgtype(*dev) cfgd(ds6000) in the production partition, we can verify that the production
IASP is no longer available. This is shown in Figure B-29.

06/23/06 08:58:54


DS6000 VARIED OFF
Figure B-29 Original IASP after switchover
3. You can now observe the toolkit status and options by entering the command dspessdta
with the IASP name DS6000, in our case, as parameter. As shown in Figure B-30, the
direction of Metro Mirror is now reversed, and the current production partition is ITCHA3,
which was originally the recovery partition.

Copy type . . . . . . . . . . : *BOTH
Request type . . . . . . . . . : 0
PPRC direction . . . . . . . . : *REVERSED
Current Production node . . . : ITCHA3
Multipath . . . . . . . . . . : *YES
Wait time . . . . . . . . . . : 60
Library . . . . . . . . . . :
Bottom
Figure B-30 dspessdta after switchover
Switch back to local site after planned outages

After a planned outage, switch back to the production system by issuing the swpprc command
in the production partition. After issuing the command, a screen displays where you can
specify how to switch back: *scheduled for planned outages or *unscheduled for unplanned
outages. Specify *scheduled, is shown in Figure B-31.

Bottom
F24=More keys
Figure B-31 swpprcback to production
During execution of this command, the following actions are performed:

1. Run the chkpprc command.
2. Vary off the IASP on the recovery system.
3. Perform Metro Mirror failover to the primary site.
4. Perform Metro Mirror failback from the primary to the secondary site.
5. Vary on the IASP on the production system.
6. By using the System i5 logical partitioning functions remove IOPs and adapters logically
from the recovery system and modify the System i5 remote partition description.
While swpprc executes, messages indicating progress with the above actions appear at the
bottom of the screen.

When the swpprc execution has completed, you can examine the toolkit log to see if
switchback to production was successful. To do this, use the viewlog command on the
production partition. An example of a log after switchback is shown in Figure B-32.

Control :
CMD ....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+
Fri Jun 23 09:38:17 2006 startDasdMgmtOperaton: First of 2 ASP records retu
Fri Jun 23 09:38:17 2006 startDasdMgmtOperaton: Record 2 of 2 ASP records r
Fri Jun 23 09:39:39 2006 addRemoveHmcDevices: Removed device at bus 2, slot
Fri Jun 23 09:39:39 2006 addRemoveHmcDevices: Removed a total of 1 device(s
Fri Jun 23 09:39:40 2006 workWithHmcProfile: FYI, device on ITCHA3 bus 2 sl
Fri Jun 23 09:39:52 2006 swpprc: SUCCESSFUL.
************End of Data********************

Figure B-32 Toolkit log after successful switchback
Check the status of the production IASP by entering the command swrkcfgsts
cfgtypw(*dev) cfgd(ds6000) on the local partition. The status should be available, as shown
in Figure B-33.

06/23/06 10:07:21


DS6000 AVAILABLE
Bottom
Parameters or command
===>
F3=Exit F4=Prompt F12=Cancel F23=More options F24=More keys
Figure B-33 Status of IASP on production after switchback
You can also check the toolkit status by entering the dspessdta command with the IASP name
as the parameter. As is shown in Figure B-34, the direction of the Metro Mirror is now normal.

Copy type . . . . . . . . . . : *BOTH
Request type . . . . . . . . . : 0
PPRC direction . . . . . . . . : *NORMAL
Multipath . . . . . . . . . . : *YES
Wait time . . . . . . . . . . : 60
Library . . . . . . . . . . :
Bottom
Figure B-34 Toolkit status after switchback to production

The application in the local partition can now resume work with the local IASP.
Re-trying SWPPRC
If for any reason executing swpprc fails before completion, you must run it again with the
*retry parameter to complete the switchover or switchback process.
Switch to remote site at unplanned outages

To perform a switch during unplanned outages, use swpprc with the parameter *unscheduled.
This command attempts to complete all the needed steps for a scheduled switch but will allow
a switch to happen even if the following errors are detected:
򐂰 Production node HMC failure
򐂰 Production Node failure
򐂰 Production DS/ESS failure (in other words, if the failback task cannot be run)
A unscheduled switch will always be an incomplete switch due to failures. The command
swpprc with parameter *complete must be run once the failures have been corrected. This will
complete Metro Mirror failover. These commands are described below.
Use the following steps to switch to the remote site in unplanned outage situations:
1. At the failure of production site, use the command swpprc *unscheduled in the remote
partition, as shown in Figure B-35.
Independent ASP name . . . . . . > DS6000 Name

Switch type . . . . . . . . . . *unscheduled *SCHEDULED, *UNSCHEDULED...
Bottom
F24=More keys
Figure B-35 swpprc*unscheduled
Shortly after the command starts executing, the message Waiting for reply to message
on message queue QSYSOPR appears, as shown in Figure B-36.

Switch type . . . . . . . . . . *unscheduled *SCHEDULED, *UNSCHEDULED...
Bottom
F24=More keys
Waiting for reply to message on message queue QSYSOPR.
Figure B-36 Message Waiting for reply
2. To reply to the message perform the following:

a. Make sure that a window with the Telnet session is open, press and hold the Shift key,
and press the Esc key. This brings up a line at the bottom of the Telnet session. Make
sure that cursor is located on this line and press Enter.
b. This brings up a screen named System Request. In this screen, specify option 6 -
Display system operator messages, as shown in Figure B-37.
System Request
System: ITCHA3
1. Display sign on for secondary job

2. End previous request
3. Display current job
4. Display messages
5. Send a message
6. Display system operator messages
7. Display work station user
80. Disconnect job
90. Sign off
Bottom
Selection
6
F3=Exit F12=Cancel
(C) COPYRIGHT IBM CORP. 1980, 2005.
Figure B-37 Display operator message

c. This brings up the screen with operator messages. In this screen confirm that you want
to perform an unscheduled switch by replying g to the message An Unscheduled
SWPPRC command was issued for IASP device DS6000? (G C), as shown in
Figure B-38. After replying to the message, press F3 twice to get back to the original
Telnet screen.
Display Messages
System: ITCHA3
Queue . . . . . : QSYSOPR Program . . . . : *DSPMSG
Library . . . : QSYS Library . . . :
Severity . . . : 99 Delivery . . . : *HOLD
Type reply (if required), press Enter.

i5/OS usage limit exceeded - operator action required.
Usage limit of 0 exceeded. Grace period expires in 34 days on 08/01/06.
Job 016703/DHQB/ANZDFTPWD5 submitted for job schedule entry ANZDFTPWD5
number 000253.
Job 016703/DHQB/ANZDFTPWD5 completed normally on 06/29/06 at 01:00:00.
i5/OS grace period expires in 34 days on 08/01/06.
An Unscheduled SWPPRC command was issued for IASP device DS6000? (G C)
Reply . . . g
Bottom
F3=Exit F11=Remove a message F12=Cancel
F13=Remove all F16=Remove all except unanswered F24=More keys
Figure B-38 Confirming Unscheduled switch by replying the message
When confirming to perform the unscheduled switch, the command executes further.
During execution, messages indicating the actions performed are displayed at the bottom
of the screen. The messages are similar to the ones shown in “Switch to remote site at
planned outages” on page 654.
3. Once the command swpprc *unscheduled has completed, use the command dspessdta
with the IASP name as the parameter to display the Metro Mirror status. Output of this
command is shown in Figure B-39. Observe that the Metro Mirror status is incomplete.
The command dspessdta is described in “Checking the Toolkit setup” on page 644.

Copy type . . . . . . . . . . : *BOTH
Request type . . . . . . . . . : 0
PPRC status . . . . . . . . . : *INCOMPLETE
Automatic PPRC Replicate . . : *NO
Multipath . . . . . . . . . . : *YES
Wait time . . . . . . . . . . : 10
Library . . . . . . . . . . :
Bottom
Figure B-39 dspessdta after unscheduled switch
You can also look at the toolkit log to check for successful execution of the swpprc
*unscheduled command. For information about how to view the log, refer to “Checking the
Toolkit setup” on page 644.
4. After the local site is operational again, perform the command swpprc *complete at the
local site to complete the switch. The command is shown in Figure B-40.

Switch type . . . . . . . . . . > *COMPLETE *SCHEDULED, *UNSCHEDULED...
Bottom
F24=More keys
Figure B-40 SWPPRC *COMPLETE

During the command execution, the message Waiting for reply to message on message
queue QSYSOPR is displayed at the bottom of the screen, as shown in Figure B-41.

Switch type . . . . . . . . . . *complete *SCHEDULED, *UNSCHEDULED...
Bottom
F24=More keys
Waiting for reply to message on message queue QSYSOPR.
Figure B-41 Waiting for reply to operator message
5. To answer the message open another telnet session with the local partition, enter the
command dspmsg qsysopr, and reply g. This is illustrated in Figure B-42.
Display Messages
System: ITCHA2
Queue . . . . . : QSYSOPR Program . . . . : *DSPMSG
Library . . . : QSYS Library . . . :
Severity . . . : 99 Delivery . . . : *HOLD
Type reply (if required), press Enter.

Cleanup of system journals and system logs successfully completed.
Cleanup has completed.
Usage limit of 0 exceeded. Grace period expires in 34 days on 08/01/06.
Job 054555/DHQB/ANZDFTPWD5 submitted for job schedule entry ANZDFTPWD5
number 000841.
Job 054555/DHQB/ANZDFTPWD5 completed normally on 06/29/06 at 01:00:00.
An Unscheduled SWPPRC command was issued for IASP device DS6000? (G C)
Reply . . : G
Perform an automated Replicate from current Production node ITCHA3 to
current Backup node ITCHA2 for IASP device DS6000? (G C)
Reply . . . g
Bottom
F3=Exit F11=Remove a message F12=Cancel
F13=Remove all F16=Remove all except unanswered F24=More keys
Figure B-42 Confirm replicate from remote site to local site
6. After the command swpprc *complete has completed, the switch to the remote site is
normally done. To check, use the command dspessdta on the local or remote partition.
Verify that the Metro Mirror direction is reversed, and that the remote partition ITCHA3 is
currently the production node. This is shown in Figure B-43.

Copy type . . . . . . . . . . : *BOTH
Request type . . . . . . . . . : 0
Automatic PPRC Replicate . . : *NO
Multipath . . . . . . . . . . : *YES
Wait time . . . . . . . . . . : 10
Library . . . . . . . . . . :
Bottom
Figure B-43 Status after switch is completed
7. You can also check the status of IASP in both the local and the remote partition by using
the wrkcfgsts command, as described in “Checking the Toolkit setup” on page 644. Check
that the IASP at the remote site is available and that the IASP at the local site is not
available (varied off).
Switchback to local site at unplanned outages

To accomplish the switchback to the local site in an unplanned outage situation, issue the
command swpprc *scheduled in the local partition. This command performs the actions
needed to establish the local partition as production, and establish Metro Mirror from the local
to the remote DS system. For instructions on how to run this command, refer to “Switch back
to local site after planned outages” on page 658.
Global Mirror for an IASP

This section describes Global Mirror operations with an Independent Auxiliary Storage Pool.
In a System i5 environment, Global Mirror is typically considered for distances greater than 50
km. Since the performance impact of Global Mirror on production is very small, this mode of
replication can be used for replicating IASPs, or an entire disk space with Boot from SAN. For
information about Global Mirror with Boot from SAN, refer to “Global Mirror for the entire disk
space” on page 682.
For more information about Global Mirror refer to Part 6, “Global Mirror” on page 241. For
more information about IASPs refer to “Independent Auxiliary Storage Pools (IASPs)” on
page 635.

This solution consists of a System i5 local partition and a System i5 remote partition grouped
in a cluster. Critical application data reside in an IASP in the local partition, the IASP
containing volumes on an external disk system. A Global Mirror relation for the IASP volumes
is established with a remote disk system. The System i5 remote partition has access to this
remote disk system. In other words, the Global Mirror secondary volumes (copy of the
production IASP volumes) can be varied on for the remote partition.
For more information about System i5 clusters refer to “Clusters” on page 633.
Note that it is only necessary that IASP on both partitions (local and remote) reside on
external storage. Still, the sysbas for each partition can be on internal disks or on external
disk systems. For more information about sysbas refer to “Independent Auxiliary Storage
Pools (IASPs)” on page 635.
The solution is implemented with the System i Copy Services Toolkit available for purchase
from System i Client Technology Center in Rochester. For more information about the Toolkit
refer to “Solution description” on page 637 and “Implementation and usage” on page 641.
The solution provides continuous availability in case of planned and unplanned outages at the
local partition:
򐂰 When a planned outage is scheduled for the local partition, the following actions are done
by the Toolkit:
a. Synchronize Global Copy on secondary volumes.
b. Vary on IASP (Global Mirror secondary volumes) at the remote partition.
c. Once IASP is available to the remote partition, start Global Copy to replicate the IASP
back to the local partition.
d. After the planned outage is finished, vary on the local IASP, which now contains the
updates that were made at the recovery site, at the local partition.
e. Start the replication again in the original direction.
򐂰 For unplanned outages the toolkit performs the following actions:
a. Ensure that Global Mirror secondary volumes are consistent and as up-to-date as
possible by performing a fast reverse restore from FlashCopy target volumes to the
Global Mirror secondary volumes.
b. Vary on IASP (Global Mirror secondary volumes) at the remote partition where
production application continues to run using this IASP.
c. When the local site is back, replicate updated IASP to the local site using a Global
Copy.
d. Vary on IASP at the local partition where production can now resume.
An overview of IASP Global Mirror is shown on Figure B-44.
Connection for cluster

Production Recovery
GM
secondary
Global IASP
Mirror
IASP Flash
copy
Figure B-44 Global Mirror of IASP
Solution benefits
Global Mirror of an IASP includes the following benefits:
򐂰 Since replication to the remote site is done asynchronously, the impact on application
response time is minimal.
򐂰 For System i5 clients who require long-distance replication, Global Mirror, thanks to its
inherent asynchronous mode, is the most suitable solution amongst other long-distance
replication solution offerings.
򐂰 Clients get the possibility to balance between RPO and bandwidth: If clients can cope with
some bigger RPO, they can use relatively narrow bandwidth for links between the two
sites. On the other hand, if they use large bandwidth, RPO will be very small and the loss
of data at recovery is almost negligible.
򐂰 Basically, recovery takes as long as is needed to vary on an IASP. This recovery is
significantly shorter with Global Mirror than with solutions relying on Boot from SAN, or an
external loadsource, where IPL is needed for recovery.
򐂰 In a production partition we run critical applications, while in a remote partition we can run
a non production workload like testing or development. This is different from
implementation using Boot from SAN where the recovery partition has to remain in
standby, typically without any workload.
򐂰 Global Mirror requires a carefully performed sequence of steps during the recovery
procedure in unplanned outage situations. For clients with a huge number of volumes, it is
virtually impossible to perform these steps manually. The System i Copy Services Toolkit,
which is used as part of the solution, provides full automation of all the tasks needed to
recover the IASP.
򐂰 Once this solution is established it requires very little maintenance and administration.
Therefore clients may prefer it to other high availability solutions available for System i5.

When clients plan for Global Mirror, their principal considerations are recovery point objective
(RPO) and needed bandwidth between local and remote DS systems. Expectations about
RPO should be well understood, and careful sizing is needed to provide enough bandwidth to
achieve the expected RPO. For sizing Global Mirror links in a System i5 environment, follow

these steps (for detailed information refer to the IBM Redbook iSeries and IBM TotalStorage:
A Guide to Implementing External Disk on Eserver i5, SG24-7120):
1. Collect system i5 performance data. The best is to collect it over one week, and if needed,
during heavy workload such as when running end-of-month jobs.
2. If the customer has already established IASP, observe in performance reports the number
of writes that go to the IASP. If the IASP is not set up yet, observe the number of writes
that go to the database:
a. Multiply the writes to IASP by the reported transfer size to get the write rate (MB/sec)
for the entire period of collecting performance data.
b. Look for the highest reported write rate. Calculate the needed bandwidth as follows:
i. You should assume 10 bits per byte for network overhead.
ii. If the compression of devices for remote links is known you may apply it. If it is not
known you may assume a 2:1 compression.
iii. You should assume a maximum 80% utilization of the network.
iv. Apply 10% uplift factor to the result to account for peaks in the 5-minute intervals.
The Client Technology Center can perform an in-depth analysis of bandwidth requirements by
using testing workload PEX, if more accurate sizing is needed.
Other planning considerations and requirements for this solution are the same as for the
solution for Metro Mirror of an IASP described in “Planning and requirements” on page 639,
with the following change: for this solution a Global Mirror license is needed for the local and
remote DS systems, and FlashCopy license is needed for the remote DS system.
Note: For information about the toolkit contact the Client Technology Center via the
http://www-03.ibm.com/servers/eserver/services/iseriesservices.html
Considerations
The same considerations as in “Considerations” on page 640 for Metro Mirror of an IASP also
apply for the Global Mirror of an IASP.
Also, once a volume is assigned to an i5/OS partition and added to that partition’s
i5/OS configuration. After the characteristic changes are made, for example, protection type,
capacity, and so on, by destroying and recreating the volume or by utilizing the DS CLI, the
volume can be reassigned to the i5/OS configuration. To simplify the configuration, we
recommend a symmetrical configuration between two IBM Storage systems, creating the
same volumes with the same volume ID that decides the LSS_ID.
This solution is implemented as a service included with System i Copy Services Toolkit. The
installation steps for the Toolkit for Global Mirror are similar to those steps explained for Metro
Mirror in “Implementation and usage” on page 671.
After the Toolkit for Global Mirror is installed and tested you are able to perform the switch to
the remote site during planned and unplanned outages. The switch to remote site and the
failback to the production site are achieved with a series of actions performed by the toolkit.
They are started by toolkit commands swpprc and falovrgmir.
The Global Mirror status and toolkit setup status can be checked by issuing the commands
chkpprc and dspessdta, respectively.
Results of toolkit actions during the switch, or for checking the status after the switch, can be
observed in the Toolkit log by using a viewlog command.
The Toolkit triggers the different tasks by communicating with the System i5 HMC, performing
System i5 commands within the partition, and communicating with external disk systems by
using the DSCLI. This is illustrated in Figure B-7 on page 642.
Checking status
For a description on how to check the IASP status, refer to “Implementation and usage” on
page 641.

To switch to the remote system at planned outages use the command swpprc *scheduled.
For information about how to enter this command refer to “Implementation and usage” on
page 641.

To switch to the remote site in case of a failure of the System i5 or the DS storage at the local
site or a disaster at local site, use the following commands:
1. In the remote partition perform the command swpprc *unscheduled. This command places
the cluster and IOPs in partitions in the appropriate status for a failover.
2. In the remote partition enter the Toolkit command falovrgmir. In the Global Mirroring
Failover screen, specify the IASP name, as shown in Figure B-45.
Global Mirroring Failover (FALOVRGMIR)
Environment name . . . . . . . . gmir Character value
Bottom
F24=More keys
Parameter ENV required.
Figure B-45 falovrgmir

While executing this command, the toolkit performs the following actions:
a. Run chkpprc command.
b. Vary off the IASP at the local partition.
c. Fail over Global Copy to the remote site.
d. Check the revertible status of the volumes at the remote site and programmatically
create the correct scripts to revert FlashCopy on remote volumes.
e. Reverse FlashCopy on remote volumes.
f. Vary on IASP on Global Mirror secondary volumes to the remote partition.
3. Once the command falovrgmir has been executed, the production application continues
to run in the remote partition with the Global Mirror copy of the IASP.
Switch back to the local site at unplanned outages

Failback to local partition is achieved by running DS CLI scripts, which are contained in the
toolkit.
Metro Mirror for the entire disk space

In this section we discuss Metro Mirror for a System i5 partition entire disk space. We also
assume that Boot from SAN (BfS) is used for both local and remote partitions.
For more information about Metro Mirror refer to Part 4, “Metro Mirror” on page 107. For more
information about Boot from SAN refer to the IBM Redbook IBM System Storage DS6000
Series: Architecture and Implementation, SG24-6781.
Because of the System i5 single level storage architecture, it is required that all disk units for
the local partition reside on external storage and that all of them are replicated (with Metro
Mirror) to the remote site.
In the solution we review here, the loadsource is an external disk, is connected through Boot
from SAN IOP, and is also replicated to the remote site using Metro Mirror. We assume that
the standby partition in the System i5 (recovery partition) at the remote site has access to the
copy of the production volumes (Metro Mirror targets), and that Boot from SAN is done from
the Metro Mirror copy of the loadsource. This solution is depicted in Figure B-46.
Production Recovery
Metro
BfS
Mirror
BfS
Figure B-46 Metro Mirror of entire disk space
In case of planned or unplanned outage at the production site, the stand-by partition at the
recovery site uses Boot from SAN to start the system from the Metro Mirror copy of
production volumes. After the boot is completed, the recovery system (stand-by partition) has
access to an exact clone of the production system, including database files, application
programs, user profiles, job queues, dataareas, and so on. Critical applications can continue
to run in this production system clone.
After planned outage is over or the unplanned outage is fixed, a Metro Mirror of all the disk
space is performed from the standby partition to the local partition. This copies back to
production system all the updates that occurred during the outage. The original local partition
can then be rebooted (Boot from SAN) from the updated primary volumes and will also get
access to primary volumes updated with the changes that accused during the outage.
For more information about System i5 single level storage refer to “Single-level storage” on
page 631. For more information about Boot from SAN refer to refer to the IBM Redbook IBM
System Storage DS6000 Series: Architecture and Implementation, SG24-6781.
Solution benefits
This solution offers the following major benefits:
򐂰 It can be implemented without any updates or changes to the production i5/OS. If for any
reason you are not ready yet to set up your application in an IASP, you can still get the
benefits of a Metro Mirror solution by applying it for the entire partition disk space.
򐂰 The solution does not require any special maintenance on the production or stand-by
system partition. Practically, the only required task is to set up the Metro Mirror relations
for all the volumes making up the partition entire disk space. Once done, no further actions
are required.
򐂰 With the Metro Mirror technology and mechanisms, the performance impact on the
production system is usually smaller than with similar solutions by other external storage

vendors. However, the minimum impact on production performance, even when using
Metro Mirror, is achieved when using IASPs.
򐂰 Since Metro Mirror works completely within DS systems, this scenario does not use any
processor or memory resources of production and remote partition in System i5. This is
different from other i5/OS replication solutions, which use some CPU resources in the
production and recovery partitions.

The following points should be taken into consideration when planning a Metro Mirror of the
entire disk space:
򐂰 It is essential to properly size primary and secondary DS storage systems.
򐂰 Careful sizing of Metro Mirror links is needed. To size the number of links use the following
steps:
a. Collect system i5 performance data. It is is to collect them over a one-week period, and
if applicable, during heavy workload like when running end-of-month jobs.
b. Multiply the writes/sec by reported transfer size to get the write rate (MB/sec) for the
entire period over which performance data were collected.
c. Look for the highest reported write rate. Size the number of Metro Mirror links so that
the bandwidth can accommodate the highest write rate.
򐂰 Since Boot from SAN is used in this solution, hardware and software requirements for
Boot form SAN need to be taken into consideration.
򐂰 We strongly recommend using two external load sources mirrored by i5/OS mirroring,
each of them connected via BfS.
For guidelines on how to size external storage for System i5, as well as for information about
how to collect System i5 performance data and which reports are needed for sizing external
storage, as well as requirements for BfS, refer to the IBM RedbookiSeries and IBM
TotalStorage: A Guide to Implementing External Disk on Eserver i5, SG24-7120.
Considerations
Some other important considerations for this solution are as follows:
򐂰 When doing a Metro Mirror of all the disk units in the local partition, not only the critical
application data but all i5/OS objects (including temporary objects, job queues, message
queues, and so on) are synchronously copied to remote site. This usually impacts
production performance more than just doing a Metro Mirror of objects in an IASP. Some
installations can experience significant differences in performance between Metro Mirror
of entire disk space and Metro Mirror of an IASP.
Also, it should be noted that Metro Mirror of the entire disk space sometimes causes the
System i5 storage management to change the transfer sizes of IO operations. The transfer
size changes can have a significant performance impact to the production parititon.
򐂰 Within the System i5 single level storage architecture, updates to application data are
performed in main memory. Updated database rows usually stay in main memory for
some time and are only written to disk as a result of a page swap. However, if a database
is journaled, changes to journaled objects are written immediately to the journal receiver
on disk.
When a switch to the remote site is performed at unplanned outages, the updates that
were still in main memory and not yet written to disk will not be reflected at the remote site,
since Metro Mirror only replicates what is on disk. However, with a database, since the
journaled objects are immediately written to disk and replicated by Metro Mirror, they will
be available at the remote site. Thus we strongly recommend journaling application
objects when using Metro Mirror. This applies whether you are mirroring the entire disk
space or just an IASP.
For more information about System i5 single level storage refer to “Single-level storage” on
page 631.
򐂰 After a remote partition is IPLed from Metro Mirror secondary volumes, it has access to an
exact clone of the production partition, including i5/OS attributes and descriptions. Usually
some changes need to be done to the cloned partition for applications to run properly on
the recovery System i5. For more information about this topic, refer to the IBM
RedbookiSeries and IBM TotalStorage: A Guide to Implementing External Disk on
Eserver i5, SG24-7120.

In this section we describe how to implement Metro Mirror for the entire System i5 disk space
and how to use it to achieve Continuous Availability in planned and unplanned outage
situations.
Implementation
Since implementation includes several System i5 related tasks, we recommend that a System
i5 specialist be involved in implementing it. The implementation is done according to the
following steps:
1. For each System i5 production and remote partitions, install FC adapters and IOPs as
needed to implement Boot from SAN.
2. On the DS system define logical volumes for the System i5 and connect them to the
System i5.
3. Install i5/OS in the local partition or migrate an existing i5/OS to external disk.
4. Restore and start System i5 production applications.
5. Start Metro Mirror for all System i5 volumes. For more information about how to do this
refer to 12.1, “Basic Metro Mirror operation” on page 114. You can use DS CLI on System
i5 to perform this action.
For more information about the preceding topics refer to the IBM Redbooks iSeries and IBM
TotalStorage: A Guide to Implementing External Disk on Eserver i5, SG24-7120, and IBM

Check tagging of external loadsource
In order to check the System I5 external loadsource, perform the following steps at the
System i5 HMC: In navigation area of the HMC GUI, expand Server and Partition and click
Server management to bring up the Server and Partition: Server Management panel. In this
panel expand Server, expand Partitions, look for the icon on the local partition, expand it,
right-click the lower icon (profile title), and select Properties from the pull-down menu. This
brings up the Logical Partition Profile Properties panel.
In this panel click the tab Tagged I/O and note the location of IOP or FC adapter for the
loadsource, as shown in Figure B-47.
Figure B-47 Tagged loadsource
On the Tagged I/O tab click the Select button at the tagged load source. This brings up the
panel Load Source Device. On the Load Source Device panel expand the unit, expand buses,
look for the slot that is tagged, select the slot, and click Properties. This is shown in
Figure B-48.
Figure B-48 External loadsource
This brings up the Physical I/O Properties panel, as shown in Figure B-49.
Figure B-49 Properties of loadsource
Click the I5/OS tab. You are presented with the description of IOP, as shown in Figure B-50.
Note the feature code 2847 that indicates Boot from SAN.
Figure B-50 IOP for Boot from SAN

Note: The examples shown below were done on a DS8000, using LSS number 37. When
performing these actions on a DS6000, another LSS number will be used since DS6000
supports only LSS numbers 00 to 1F. Otherwise, the DS CLI scripts used for DS6000 are
the same as shown in these examples.
To achieve a switch to remote site perform the following steps:

1. Power down the local System i5 partition. This ensures that all the data are written to disk
and that the remote partition IPL will be normal.
Note: A normal IPL in System i5 occurs after the system is powered down with the
power down System (PWRDWNSYS) command and no jobs have ended abnormally.
All other IPLs are abnormal for i5/OS. An abnormal IPL takes longer because of
additional recovery and verification needed.
2. By using the DS GUI or DS CLI, list the status of all the volumes in the local partition and
the recovery partition. The DS CLI command and a sample result are shown in
Figure B-51. Observe that both Metro Mirror primary and secondary volumes are in Full
Duplex state.

Date/Time: June 27, 2006 1:21:56 PM CEST IBM DSCLI Version: 5.1.600.260 DS: IBM
2107-7503461
ID State Reason Type SourceLSS Timeout (secs) Critical Mod First Pass Status
=================================================================================================
Figure B-51 LSPPRC
Note that n the example shown below we used the same scripts to perform actions on
primary and secondary DS systems. This can be done since the volumes on primary and
secondary DS have the same volume IDs. The action was performed on one or the other
DS by specifying relevant DS CLI profile as part of the DS CLI command. Each profile
contains IP addresses and image IDs of the local and remote DS storage systems.
Note: The DS CLI commands and scripts can be used from a System i5. To do this,
you need a separate partition in the System i5 (not the production partition or recovery
partition). The separate partition must have connectivity to both the local and remote
DS systems.
3. Suspend Metro Mirror for the local partition volumes. The DS CLI script for suspending the
volumes is shown in Figure B-52.
#
# suspend Metro Mirror
#
pausePPRC 3700-3707:3700-3707
Figure B-52 Script for suspend Metro mirror
Performing the script and the results are shown in Figure B-53. Suspend Metro Mirror is
performed on the primary DS system.
C:\Program Files\ibm\dscli>dscli -cfg "c:\R1_dscli.profile" -user admin -script

"c:\PausePPRC.cli"
Date/Time: June 27, 2006 2:01:40 PM CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7503461
Figure B-53 Performing script for suspend Metro Mirror
4. Perform a Metro Mirror failover to the secondary DS system. For more information about
failover refer to 12.3, “Failover and Failback” on page 115. The DS CLI script for failover
and the results of running the script are shown in Figure B-54 and Figure B-55. Failover is
performed on a secondary DS system.
#
# Residency Mainz - failover Metor Mirror #
failoverPPRC -type mmir 3700-3707:3700-3707
Figure B-54 Script for failover of Metro Mirror

"c:\Failover.cli"
Figure B-55 Performing script for Failover of Metro Mirror
Observe that both primary volumes (volumes belonging to the production partition) and
secondary volumes (belonging to the recovery partition) are in Source suspended state,
as shown in Figure B-56 and Figure B-57 on page 680.
Note that in our examples the DS primary system is 7503461 and the DS secondary
system is 7520781.

ID State Reason Type SourceLSS Timeout (secs) Critical mode First Pass Status
===================================================================================================
3700:3700 Suspended Host Source Metro Mirror 37 300 Disabled Invalid
Figure B-56 Status of primary volumes after suspend and failover

dscli> lspprc 3700-3707:3700-3707
=====================================================================================================
Figure B-57 Status of secondary volumes after suspend and failover
5. IPL the remote partition from the secondary volumes. To perform this, the secondary
volumes have to be connected to the standby partition. We recommend that the standby
partition contain two Boot form SAN (BfS) IOPs, each one of them connected to a copy of
a mirrored mate of load source. Make sure that one of the BfS IOPs is tagged as load
source in the System i5 HMC.
To IPL the remote partition perform the following steps from the System i5 HMC:
a. In the HMC navigation area expand Server and Partition and click Server
management to bring up the Server and Partition: Server Management panel. In this
panel expand Server, expand Partitions, and look for the remote partition icon.
b. Expand the view by clicking the plus sign next to the icon, then right-click the upper
icon and select Activate from the pull-down menu. This is shown on Figure B-58.
Figure B-58 IPL partition
6. Once the remote partition is up and running, the applications can continue to run in the
recovery partition.
7. When the planned outage is finished, perform Metro Mirror failback to the primary volumes
at the production site. Doing this ensures that updates done at the remote partition during
the outage are now replicated back to the production partition. You may want to do a
Metro Mirror failback even before the outage is over, depending on the type of outage. The
DS CLI script used for the failback and performing the script are illustrated in Figure B-59
on page 681 and Figure B-60. Note that the failback is done on the secondary DS system.
#
# Residency Mainz - failback MetrorMirror
#
failbackPPRC -type mmir 3700-3707:3700-3707
Figure B-59 Script for failback of Metro Mirror

"c:\Failback.cli"
Figure B-60 Performing script for failback of Metro Mirror
Fail back to local partition

To switch back (fail back) to the local partition perform the following steps:
1. Power down the recovery partition.
2. Suspend the Metro Mirror on secondary volumes. The relevant DS CLI script and results
of its execution are shown in Figure B-52 on page 678 and Figure B-61, respectively. This
action is performed on the secondary DS system.

"c:\PausePPRC.cli"
Figure B-61 Pause Metro Mirror or secondary DS
3. Fail over Metro Mirror to the primary volumes at the production site. The DS CLI script for
this action and results of its execution are shown in Figure B-54 on page 679 and
Figure B-62. This action is performed on the primary DS system.

"c:\Failover.cli"
Figure B-62 Fail over to the primary DS

4. Fail back Metro Mirror from primary volumes to secondary volumes. The DS CLI script and
result of its execution are shown in Figure B-63. This is performed on the production DS
system.

"c:\Failback.cli"
Figure B-63 Failback on primary DS
5. IPL the production parititon and start the applications at the production site. Make sure
that the correct IOP for Boot from SAN is tagged for load source on the System i5 HMC,
as described earlier in this section.

In unplanned outage situations like a System i5 or DS storage system failure at the
production site, perform the following steps to switch to the remote site:
1. Perform Metro Mirror failover to the secondary volumes at the remote site, as is described
in earlier in this section.
2. Make sure that the secondary volumes are connected to the recovery partition, and that
the correct BfS IOP is tagged as load source in System i5 HMC at the recovery site.
Perform an IPL of the recovery partition. Since the local partition was probably not
powered down, IPL at the recovery site will be abnormal.
After the remote partition is IPLed, it becomes a clone of the local partition, and the
production applications continue to run with this cloned system. For considerations
regarding the cloning, like different serial numbers, i5/OS licenses, and others, refer to the
IBM RedbookiSeries and IBM TotalStorage: A Guide to Implementing External Disk on
Eserver i5, SG24-7120.
3. After the failure on production site is fixed perform Metro Mirror failback to primary
volumes. If this is not possible remove Metro Mirror or secondary DS system, and
establish Metro Mirror from secondary to primary.
Global Mirror for the entire disk space

In this section we look at doing a Global Mirror of the entire local partition disk space. This
solution provides continuous availability with a recovery site located at a long distance while
minimizing the production performance impact. For more information about Global Mirror
refer to Part 6, “Global Mirror” on page 241.
In this solution, the entire local partition disk space resides on external storage. The load
source is also external and connected to the System i5 via Boot from SAN (BfS). A Global
Mirror for all volumes belonging to the production partition is established with another DS
storage system located at the remote site.
An important consideration here is the balance between the bandwidth to be provided for
Global Mirror, and RPO. It is essential to understand this issue and set the right expectations
when planning for this solution.
For more information about System i5 Boot from SAN refer to the IBM Redbook IBM System
Storage DS6000 Series: Architecture and Implementation, SG24-6781.
In planned or unplanned outage situations, a standby partition in the System i5 at the remote
site is IPLed from the Global Mirror secondary volumes. The Global Mirror copy of the load
source is also connected to the standby partition with Boot from SAN.
To ensure that the data to recover from are as up-to-date as possible, the solution includes
FlashCopy volumes of the Global Mirror targets. A fast restore from FlashCopy targets of the
Global Mirror secondary volumes is performed before IPL.
The solution is depicted in Figure B-64.
Production Recovery
GM secondary
Global
BfS
Mirror Flashcopy tgts
BfS
Figure B-64 Global Mirror of entire disk space
Solution benefits
Some of the benefits of establishing a Global Mirror for the entire disk space include:
򐂰 The solution provides replication of production data over long distances while minimizing
production performance impact.
򐂰 The solution offers the possibility to balance between RPO and bandwidth. If you can cope
with longer RPO, you can use relatively narrow bandwidth for links between the two sites.
On the other hand, if you have a large bandwidth, the RPO will be very small and of data
loss at recovery will be almost negligible.
򐂰 Considerations about impact on System i5 production performances which apply to Metro
Mirror of entire disk space do not apply to Global Mirror of entire disk space. Therefore this
solution is recommended to System i5 customers. For more information about Metro
Mirror impact on System i5 performances refer to “Considerations” on page 674.
򐂰 The solution does not require any special maintenance on the production or standby
partition. Practically, the only required task is to set up Global Mirror for the entire disk
space.
򐂰 Since Global Mirror is entirely driven by the DS storage systems, this solution does not
use any processor or memory resources from the System i5 production and remote

partition. This is different from other i5/OS replication solutions, which use some of the
production and recovery partitions resources.

Following points should be taken into account when planning for Global Mirror of entire disk
space:
򐂰 It is very important to carefully size bandwidth of connection links between production and
recovery site. Proper sizing of bandwidth can only be done based on System i5
performance data. Sizing guidelines and instructions on how to collect System i5
performance data, which are given in “Planning and requirements” on page 669, are valid
also for Global Mirror of the entire disk space. The only difference being: when sizing
Global Mirror for entire disk space you take into account all reported writes/sec, you do not
apply percentage of writes that go to IASP.
򐂰 You must also properly size the primary and secondary DS systems.
򐂰 Since Boot from SAN is used in this solution, hardware and software requirements for
Boot from SAN need to be taken into consideration.
򐂰 We strongly recommended that you use two external load sources mirrored by i5/OS
mirroring, each of them connected via BfS.
For more information about the preceding topics you can refer to the IBM Redbook iSeries
and IBM TotalStorage: A Guide to Implementing External Disk on Eserver i5, SG24-7120.
Considerations
Other considerations for this solution are:
򐂰 Recovery from Global Mirror secondary volumes in unplanned outage situation requires
additional steps compared to the recovery from Metro Mirror. The extra steps are required
to ensure a minimal data loss. In environments with large capacity and thus a large
number of volumes, it is extremely difficult to perform these steps by only issuing DS CLI
commands, and the use of an automation tool is recommended.
򐂰 We also recommend using i5/OS journaling with this solution. For more information about
journaling, refer to “Considerations” on page 674.
Customers who want to implement Global Mirror for the entire System i disk space may use
the System i Copy Services Toolkit available for purchase from the System i Client
Technology Center in Rochester. The toolkit is described in “Global Mirror for an IASP” on
page 667. Once the Toolkit is installed, you can do a switchover to the remote site during
planned and unplanned outages by using the Toolkit commands.
Still, for smaller installations with only a few volumes, it is feasible to perform the switchover to
the remote site by simply using the DS CLI commands. We describe the steps to perform for
both situations: either using Toolkit or using DS CLI.
For information about how to set up Global Mirror refer to Part 6, “Global Mirror” on page 241.
Switch to the remote site at planned outages

First we describe the sequence of steps when using the Toolkit and then using the DS CLI.
Using the Toolkit

When using the Toolkit, follow these steps:
1. Power down the System i5 local partition to ensure that all data are written to disk and that
the IPL for the remote partition will be normal. For more information about System i5 disk
space and IPL refer to “Single-level storage” on page 631 and “Switch to remote site at
planned outages” on page 671.
2. Use the toolkit command swpprc to fail over to the remote partition.
3. Use the toolkit command chghostvol to connect Global Mirror secondary volumes to the
remote partition. With this command you activate DS CLI scripts to connect volumes to FC
adapters in the System i5 remote partition.
4. Check that the external load source is tagged in the remote System i5 HMC and perform
an IPL of the remote partition. For information about how to achieve this refer to “Switch to
remote site at planned outages” on page 678.
If customers want all the steps listed to be performed automatically by one toolkit command,
they can request this as an additional service with the toolkit services.
Using DS CLI
Note: The examples shown below were done on a DS8000, using LSS numbers 37 and
39. When performing these actions on a DS6000, another LSS number will be used since
the DS6000 supports only LSS numbers 00 to 1F. Otherwise, DS CLI scripts used on the
DS6000 are the same as shown in these examples.
Perform the following steps to switch to the remote site:

1. Power down the System i5 local partition by issuing the command pwrdwnsys in a Telnet
session. For more information about how to use IBM personal communication to Telnet to
the System i5, refer to iSeries Information Center located at the following Web page:

2. Pause Global Mirror on the local DS storage system, as shown in Figure B-65.
dscli> pausegmir -lss 37 -session 01

Date/Time: July 4, 2006 3:45:01 PM CEST IBM DSCLI Version: 5.1.600.260 DS:
IBM.2
107-7503461
Figure B-65 Pause GM
3. Pause Global Copy on the local system, as shown in Figure B-66.
dscli> pausepprc 3700-3707:3700-3707

Figure B-66 Pause Global Copy
4. On remote DS perform failover of Global Copy, as shown in example on Figure B-67.
dscli> failoverpprc -type gcp 3700-3707:3700-3707

Figure B-67 Failover Global Copy
5. In planned outage situations, power down the local partition and suspend Global Mirror
before the failover. This eliminates the need to perform a fast restore of the FlashCopy.
6. IPL the remote partition from the Global Mirror secondary volumes using Boot from SAN.
For instruction on how to do this refer to “Switch to remote site at planned outages” on
page 678. Once the partition is IPLed, it will be a clone of the local partition, and
production work can continue in the cloned partition at the remote site.
Fail back to local site at planned outages

To do a failback to the local site at planned outages, use the steps described below. We
describe the steps when using the toolkit and when using DS CLI successively.
Using the toolkit

The Toolkit provides DS CLI scripts for failback to the local site after a planned outage. The
scripts reside at the remote partition.
Using DS CLI
To use this:
1. On the remote DS storage system, perform a failback, as shown in Figure B-68.
dscli> failbackpprc -type gcp 3700-3707:3700-3707

Figure B-68 Failback Global Copy
2. On the local DS storage system, perform a failover, as shown in Figure B-69.

Figure B-69 Fail over to local site
3. IPL the local partition.

4. Perform a failback to the remote site, as shown in Figure B-70.
dscli> failbackpprc -type gcp 3700-3707:3700-3707

Figure B-70 Failback Global Copy
5. On local DS resume Global Mirror, as shown in Figure B-71.
dscli> resumegmir -lss 37 -session 01

Date/Time: July 4, 2006 4:21:12 PM CEST IBM DSCLI Version: 5.1.600.260 DS:
IBM.2
107-7503461
Figure B-71 Resume Global Mirror
Switch to the remote site at unplanned outages

To achieve the switch to the remote site at unplanned outages use the steps described
below. We describe the steps when using toolkit and when using DS CLI.

Using the Toolkit
To switch over to the remote site at unplanned outages, perform the following steps:
1. Use the toolkit command falovrgmir to fail over to the remote partition. During this
command execution, the following actions are taken:
a. Run the command chkpprc to check for the correct status of the Global Mirror.
b. Fail over Global Copy to the remote site.
c. Check the revertible status of the volumes at the remote site and programmatically
create the correct scripts to revert the FlashCopy on remote volumes.
d. Reverse FlashCopy on remote volumes.
2. Use the Toolkit command chghostvol to connect Global Mirror secondary volumes to the
remote partition. With this command you activate DS CLI scripts to connect volumes to FC
adapters in the remote partition.
3. Check that the external load source is tagged at the remote system i5 HMC and perform
an IPL of the remote partition. For information about how to achieve this, refer to “Switch to
remote site at planned outages” on page 678.
If customers want all the steps listed to be performed automatically by one toolkit command,
they can request this as an additional service with the Toolkit services.
Using the DS CLI

To use this:
1. After failure of the local System i5 or the DS storage system, suspend or stop Global
Mirror on the local DS, if possible. We recommend checking the status of Global Copy and
paths, as well.
2. Fail over Global Copy on the remote DS, as shown in Figure B-72.

Figure B-72 Failover Global Copy
3. List FlashCopy volumes to see the status of sequences and revertible volumes. The DS
CLI command and partial output are shown in Figure B-73.
dscli> lsflash -l -fmt default 3700-3707:3900-3907

ID SrcLSS SequenceNum Timeout ActiveCopy Recording Persistent Revertible SourceWriteEnabled Target
==========================================================================================================
3700:3900 37 44AA7EA4 300 Disabled Enabled Enabled Disabled Enabled Disabled
dscli>
Figure B-73 LsFlash at GM unplanned outages
Examine the sequence numbers and volumes revertible status, and consequently decide
to revert FlashCopy, commit FlashCopy, or not take any action. For more information refer
to Chapter 21, “Global Mirror options and configuration” on page 263. In our example all
sequence numbers are equal and we do not have revertible volumes, so we do not
perform any action here.
4. Reverse FlashCopy on remote DS, as shown in Figure B-74
dscli> reverseflash -fast -tgtpprc 3700-3707:3900-3907

Figure B-74 Reverse FlashCopy
5. IPL the remote partition from the Global Mirror secondary volumes by using BfS. Once IPL
is done, the production application can continue to run in the clone of the local partition.
FlashCopy of IASP
The downtime needed to save application data to tape is critical for many System i5
installations because saving to tape is usually a part of typical batch jobs running in those
environments. For many clients, this downtime window is generally very limited.
The solution described in this section is based on FlashCopy and ensures minimal downtime
for saving application data to tape.
For more information about FlashCopy, refer to Part 3, “FlashCopy” on page 35. For more
information about IASP refer to “Independent Auxiliary Storage Pools (IASPs)” on page 635.
The setup for this solution consists of two System i5 partitions. Usually both partitions are
defined in the same physical System i5, but they could also reside in two separate units.

The two partitions are grouped in a cluster. We call the production partition, the partition
where the production applications run. The other partition is called the backup parititon and
typically clients would run testing or development workload in the backup partition. Both
partitions are connected to the same DS system. The production application runs in an IASP
that resides in the DS system and contains FlashCopy primary volumes.
To take a backup of the application using the backup partition, follow these steps:
1. Perform a FlashCopy of the IASP.
2. Vary on the IASP FlashCopy targets in the backup partition.
3. Use the backup partition to save the data to tape without impacting the production
partition.
During the backup operation, the production application continues to run in production
partition.
We recommended varying off the IASP in the production partition just before taking the
FlashCopy. Doing so will minimize the time needed to bring up the application in the backup
partition.
The solution is depicted in Figure B-75.
Production IASP
Flash
copy
Backup IASP
Backup
Save
Figure B-75 FlashCopy of an IASP
The solution is implemented through the System i Copy Services Toolkit.
Note: For information about the Toolkit contact the Client Technology Center via the
http://www.ibm.com/servers/eserver/services/iseriesservices.html
Solution benefits
The most important benefits are as follows:
򐂰 The production application downtime is only is as long as it takes to vary off the IASP,
perform a FlashCopy of the volumes in the IASP, and vary on IASP. Usually this time is
estimated to be about 30 minutes. This is very little compared to the downtime normally
experienced when saving to tape without a Save While Active function.
For more information about Save While Active refer to the iSeries Information Center on
the following Web page:
http://publib.boulder.ibm.com/iseries
򐂰 The performance impact on the production application during the save to tape operation is
minimal since it is only influenced by the FlashCopy activity, which is mostly confined
within the DS system. Furthermore, this FlashCopy activity is typically very low since save
to tape operations are typically performed at night or during light workload periods.
򐂰 This solution can be implemented together with Backup, Recovery and Media Services for
iSeries (BRMS), a System i5 software for saving application data to tape.
For more information about BRMS refer to the iSeries Information Center at the following
web page:

The following points should be taken into account when planning for this solution:
򐂰 An important requirement is the need to run the application in an IASP. If you do not have
your applications in IASPs yet, you must do so before installing the System i Copy
Services Toolkit (you can also engage IBM services to set up your application for IASP).
򐂰 A solution that uses the System i Copy Services Toolkit must be approved by the Client
Technology Center.
򐂰 Each IASP has its own Fibre Channel attachment cards. This is valid for both System i5
and earlier System i models like 8xx and 270.
򐂰 A FlashCopy license must be purchased for the DS or ESS system using this solution.
򐂰 For System i5 software prerequisites, the i5/OS software for clustering (5722-SS1 option
41) must be installed in both production and recovery partition. This prerequisite is needed
on each System i5 and earlier System i models like 8xx and 270..
Considerations
This solution requires you to vary off the production IASP every time a FlashCopy of the IASP
is taken. Some clients prefer not to vary off the IASP before FlashCopy, and therefore may
experience a longer time to vary on the IASP in the backup partition.
Other considerations listed under “Considerations” on page 640 for Metro Mirror of IASP also
apply here.

Using Metro Mirror and FlashCopy of IASP in the same scenario

Many clients like to combine their disaster recovery solutions with minimizing their backup
window. In this case, they implement Metro Mirror or Global Mirror of an IASP along with
FlashCopy of the IASP. Some may decide to use FlashCopy on the local DS, while others
may decide to use FlashCopy on the remote DS and implement saving to tape at the remote
site.
The following should be taken into account when planning and implementing such combined
solutions:
򐂰 System i Copy Services Toolkit for both Metro Mirror (or Global Mirror) and FlashCopy
should be purchased.
򐂰 If FlashCopy is implemented at the remote site, the solution typically needs two remote
System i5 partitions: one for recovery from Metro Mirror (or Global Mirror) targets and one
for FlashCopy. Each of them requires dedicated FC adapters.
򐂰 Regardless of whether FlashCopy is implemented at the local or remote site, it requires
that the production IASP be varied off before taking the FlashCopy. For more information
refer to “Considerations” on page 691.

The solution is implemented through System i Copy Services Toolkit. Installation of the toolkit
code, creating the DS CLI script, and testing are performed as part of the toolkit-provided
services.
The toolkit installation steps are described in “Implementation and usage” on page 641.
Once the toolkit is installed and tested, the following actions are performed to check the
solution components and perform a save to tape in the backup partition.
Checking solution components

You may want to check the following solution components:
򐂰 IASP status
򐂰 FlashCopy status
򐂰 Toolkit setup
򐂰 DS CLI scripts
Refer to “Checking the solution components” on page 642. To check the FlashCopy status
use the command dspessdta.
Prepare for save

To prepare the partition for taking a backup (that is, saving to tape the production application
data), use the toolkit command prpiaspbck, as described below.
In a Telnet session with the System i5 partition, enter the command prpiaspbck. You are
presented with a screen where you can insert the IASP name, as shown in Figure B-76.
Press Enter to start executing the command.
Prepare for IASP Backup (PRPIASPBCK)
Bottom
F24=More keys
Figure B-76 PRPIASPBCK
While executing the prpiaspbck command, the toolkit performs the following:
򐂰 Vary off the production IASP.
򐂰 Make a FlashCopy of the volumes in the DS system by running a DS CLI script in the
backup partition.
򐂰 Add IOPs and FC adapters to the backup partition that will attach the FlashCopy of the
production IASP volumes. For more information about IOPs refer to “Input Output
Processors” on page 632.
򐂰 Vary on the production IASP.
򐂰 Vary on the FlashCopy of the production IASP in the backup partition.
After completion of the prpiaspbck command, check the IASP status in the backup partition.
For information about how to check this refer to “Checking the solution components” on
page 642.
Saving to tape
At this stage the backup partition is ready for taking a backup of the production application
based on the copy (FlashCopy) of the production IASP.
The actual backup can be done with BRMS or by using i5/OS commands.
For more information about System i5 backups refer to iSeries Information Center at the
following Web page:
You can also consult the manual IBM Systems - iSeries Backup and Recovery Version 5
Revision 4, SC41-5304-08.
For more information about BRMS refer to the iSeries Information Center and the manual IBM
Systems - iSeries Backup Recovery and Media Services for iSeries Version 5,
SC41-5345-05.

Note that if BRMS is used to take backups in this solution, BRMS systems in the production
and backup partition must be in a BRMS Network.
Backups taken in the backup partition can be restored to the production partition. Single
System i5 objects from these backups can be restored to production partition, if needed.
Cleaning up the backup partition after a backup

After a backup is finished certain cleaning activities are required in the backup parititon to
make sure that it is in an appropriate state for the next backup operation. These activities can
be controlled and executed by the toolkit by issuing the command rsmiapsbck in the backup
partition.
In a Telnet session established with the System i5 partition, enter the command rsmiaspbck.
You are presented with a screen where you can insert the IASP name, as shown in
Figure B-77. Press Enter to start executing the command.

System: ITCHA3
1. User tasks
2. Office tasks
5. Programming
6. Communications
8. Problem handling
9. Display a menu
90. Sign off
===> rsmiaspbck

Figure B-77 Executing the command rsmiaspbck
While the rsmiaspbck command execution takes place, the following actions are performed
by the toolkit:
򐂰 Vary off the FlashCopy of the production IASP, which is attached to the backup node.
򐂰 Remove the FlashCopy for the volumes in the DS unit by running the dscli rmflash
script.
򐂰 IPL the backup node.
Once rsmiaspbck has completed its execution, the backup partition is in the appropriate state
for the next backup through the prpiaspbck command.
FlashCopy of the entire disk space
In this solution, FlashCopy is implemented for the entire disk space attached to a System i5
parititon. This solution will minimize the production partition downtime for taking backups. For
more information about FlashCopy refer to Chapter 5, “FlashCopy overview” on page 37.
The setup for this solution consists of two System i5 partitions. Usually both partitions reside
on the same physical System i5, but they could be also on different units. We call the
production partition the partition where the production applications run. The other partition is
called the backup parititon. The backup partition is standby and typically does not run any
workload. All disks attached to the production partition reside on the external DS storage
system and Boot from SAN is implemented.
The standby backup partition also contains Boot from SAN IOP.
For more information about Boot from SAN refer to the IBM Redbook IBM System Storage
DS6000 Series: Architecture and Implementation, SG24-6781.
To take a backup, perform the following steps:

1. Power down the production partition to ensure that all data is flushed out to disks and that
a subsequent IPL in the backup partition will be normal.
2. Do a FlashCopy of all production volumes.
3. IPL the production partition.
4. IPL the backup partition off the FlashCopy target volumes by using BfS.
After being IPLed, the backup partition is a clone of the production partition. Any backup
taken in the backup parititon will reflect actual and current production data, and later it will be
possible to restore this backup to the production partition.
For more information about the previous considerations, refer to the IBM Redbook iSeries and
IBM TotalStorage: A Guide to Implementing External Disk on Eserver i5, SG24-7120.

The solution is shown in Figure B-78.
Production
BfS
Flash
copy
Backup
Backup
BfS
Save
Figure B-78 FlashCopy of entire System i5 disk space
Solution benefits
This solution offers the following major benefits:
򐂰 The production application downtime is only is as long as it takes to power down the
production partition, take a FlashCopy of the production volumes, and IPL the production
partition (IPL is normal). Usually, this time is much shorter than the downtime experienced
when saving to tape without a Save While Active function.
For more information about Save While Active refer to the iSeries Information Center on
򐂰 The performance impact on the production application during the save to tape operation is
minimal since it is only influenced by the FlashCopy activity, which is mostly confined
within the DS system. Furthermore, this FlashCopy activity is typically very low since save
to tape operations are normally performed at night or during light workload periods.
򐂰 This solution can be implemented together with Backup, Recovery, and Media Services for
iSeries (BRMS), a System i5 software for saving application data to tape.
For more information about BRMS refer to the iSeries Information Center at the following
Web page:
It is important to adequately size the primary and secondary DS systems. Also, since Boot
from SAN is used in this solution, hardware and software requirements for Boot form SAN
need to be taken into consideration for both the production and backup partitions.
For guidelines on how to size external storage for System i5 as well as more information
about requirements for BfS refer to the IBM Redbook iSeries and IBM TotalStorage: A Guide
to Implementing External Disk on Eserver i5, SG24-7120.
Considerations
This solution requires powering down the production partition each time a FlashCopy is
taken, and then IPLing the partition after completion of the FlashCopy. This procedure is
strongly recommended in order to ensure a correct IPL of the backup partition later in the
process.
Some installations cannot tolerate a daily IPL of their i5/OS production system, and
consequently cannot stick to the recommended procedure. In this case we suggest that they
implement IASP and use the solution described in “FlashCopy of IASP” on page 689.

In this section we describe how to implement FlashCopy of entire disk space and use it for
taking backups of production partition.
Implementation
Since the implementation of this solution implies several System i5 related tasks, we
recommend involving a System i5 specialist.
The solution is implemented through the following steps:

1. In the production and backup partitions, install FC adapters and IOPs as needed to
implement Boot from SAN.
2. On the DS system define logical volumes for the System i5 and connect them to the
System i5.
3. Install i5/OS in the local partition or migrate an existing i5/OS to external disk. Restore and
start System i5 production applications.

For more information about the preceding topics refer to the IBM Redbooks iSeries and IBM
TotalStorage: A Guide to Implementing External Disk on Eserver i5, SG24-7120, and IBM
Check Boot from SAN tagging

Check that the correct IOPs or FC adapters are tagged as load source devices, in each of the
production and backup partitions.
Cloning the production partition to a backup partition

In the example below, we use FlashCopy with the nocopy option.
Note that if BRMS is used for saving to tape from the backup partition, some additional steps
are required to ensure that backups will correctly restore in the production partition. These
steps are described in the IBM Redbook iSeries and IBM TotalStorage: A Guide to
Implementing External Disk on Eserver i5, SG24-7120.
To clone the production parititon, proceed with the steps below.
Note: The examples shown here were done on a DS8000, using LSS numbers 37 and 39.
When performing these actions on a DS6000, another LSS number will be used since the
DS6000 supports only LSS numbers 00 to 1F. However, DS CLI scripts used on the
DS6000 are the same as those shown in these examples.
The steps are:

1. Power down the production partition by issuing the command pwrdwnsys in a Telnet
session. For more information about how to use IBM personal communication to Telnet to
a System i5, refer to iSeries Information Center on the following Web page:
2. Perform a FlashCopy of the production volumes. To do this you can either run a DS CLI
script in a Windows environment or directly execute the DS CLI script on the System i5.
When run from the System i5, the script must reside in a third, separate partition (that is,
not the production partition or the backup partition).
A script for making the FlashCopy and the command to invoke it in Windows, as well as
the output of the command, is shown in Example B-1 and Example B-2.
Example: B-1 Script for making FlashCopy

#
# - mkflash
#
mkflash -nocp 3700-3707:3900-3907
Example: B-2 Make FlashCopy

C:\Program Files\ibm\dscli>dscli -cfg "c:\R2_dscli.profile" -script "c:\R2_mkflash.cli" -user
admin
3. After the FlashCopy is performed, you may want to list the FlashCopy relationship of the
production volumes. The DS CLI script to perform this and the command to invoke the
script in Windows, as well as the output of the command, are shown in Example B-3 and
Example B-3.
Example: B-3 Script for list FlashCopy

# lsfalsh
#
lsflash 3700-3707:3900-3907
Example: B-4 lsflash

C:\Program Files\ibm\dscli>dscli -cfg "c:\R2_dscli.profile" -script "c:\R2_lsflsh.cli" -user admin
Date/Time: July 4, 2006 10:52:39 AM CEST IBM DSCLI Version: 5.1.600.260 DS: IBM 2107-7520781
SourceWriteEnabled TargetWriteEnabled BackgroundCopy
===============================================================================
====================================================
3700:3900 37 0 300 Disabled Disabled Disabled Disabled
Enabled Enabled Disabled
4. IPL the production partition and start the production applications. For instructions on how
to do this refer to “Switch to remote site at planned outages” on page 678.
5. Next IPL the backup partition from the FlashCopy target volumes, via BfS. Once the
partition is IPLed, save the entire system or specific objects to tape using BRMS or native
i5/OS commands.
For more information about System i5 backups, refer to the iSeries Information Center on
You can also consult the IBM Systems - iSeries Backup and Recovery Version 5
Revision 4, SC41-5304-08.

6. Once the backup is taken, remove the FlashCopy relationship. The DS CLI script to
perform this and the command to invoke the script in Windows, as well as the output of the
command, are shown on Example B-5 and Example B-6.
Example: B-5 Script for rmflash

# rmflash
#
rmflash -quiet 3700-3707:3900-3907
Example: B-6 Remove FlashCopy

C:\Program Files\ibm\dscli>dscli -cfg "c:\R2_dscli.profile" -script "c:\R2_rmflash.cli" -user admin
The system is now ready for the next backup.
C
Appendix C. Simple Network Management

Protocol (SNMP)
This appendix describes SNMP traps that are sent out in a remote copy environment. This
appendix repeats some of the SNMP trap information that is available in the IBM Redbook
The IBM TotalStorage DS8000 Series: Implementation, SG24-6786.

SNMP overview
The DS8000 sends out SNMP traps when a state change in a remote copy services
environment occurs. Therefore 13 traps are implemented. The traps 1xx are sent out for a
state change of a physical link connection. The 2xx traps are sent out for state changes in the
logical copy services setup.
The DS HMC can be set up to send SNMP traps to up to two defined IP addresses. eRCMF
(see Chapter 36, “IBM TotalStorage Rapid Data Recovery for UNIX and Windows” on
page 575) is listening to SNMP traps of the DS8000. Also, a Network Management program,
like Tivoli NetView®, can be used to catch and process the SNMP traps.
Physical connection events

With the trap 1xx range, a state change of the physical links will be reported. The trap is sent
out if the physical remote copy link is interrupted. The link traps will be sent from both
DS8000s, the primary and the secondary system. The PLink and SLink columns are in use
for the ESS.
If one or several links of multiple links are interrupted, a trap 100, as shown in Example C-1, is
posted and will indicate that the redundancy is degraded. The RC column in the trap will
represent the reason code for the interruption of the link. The reason codes are listed in
Table C-1 on page 704.
Example: C-1 Trap 100: PPRC links degraded

PPRC Links Degraded
UNIT: Mnf Type-Mod SerialNm LS
PRI: IBM 2107-922 75-20781 12
SEC: IBM 2107-9A2 75-ABTV1 24
Path: Type PP PLink SP SLink RC
1: FIBRE 0143 XXXXXX 0010 XXXXXX 15
2: FIBRE 0213 XXXXXX 0140 XXXXXX OK
If all links are interrupted, a trap 101, as shown in Example C-2, is posted. This event is
indicating that no communication between primary and secondary system is possible any
more.
Example: C-2 Trap 101: PPRC links down

PPRC Links Down
PRI: IBM 2107-922 75-20781 10
SEC: IBM 2107-9A2 75-ABTV1 20
Once the DS8000 can communicate again via any of the links, trap 102, as shown in
Example C-3, is sent once the interrupted links are available again.
Example: C-3 Trap 102: PPRC links up

PPRC Links Up
PRI: IBM 2107-9A2 75-ABTV1 21
SEC: IBM 2107-000 75-20781 11
Appendix C. Simple Network Management Protocol (SNMP) 703

Table C-1 PPRC path reason codes
Reason Code Description
00 No path.
01 ESCON path established.
02 Initialization failed. ESCON link reject threshold exceeded when attempting to send ELP or RID
frames.
03 Time out. No reason available.
04 No resources available at primary for the logical path establishment.
05 No resources available at secondary for the logical path establishment.
06 Secondary CU Sequence Number or Logical Sub-system number mismatch.
07 Secondary CU SS ID mismatch or failure of the I/O that collects secondary information for validation.
08 ESCON link is offline. This is caused by the lack of light detection coming from a host, peer, or switch.
09 Establish failed but will retry when conditions change.
0A The primary control unit port or link cannot be converted to channel mode since a logical path is
already established on the port or link. The establish paths operation will not be retried within the
control unit automatically.
0B Reserved for use by StorageTek™.
10 Configuration Error. The source of the error is one of the following:

򐂰 The specification of the SA ID does not match the installed ESCON adapter cards in the primary
controller.
򐂰 For ESCON paths, the secondary control unit destination address is zero and an ESCON
Director (switch) was found in the path.
򐂰 For ESCON paths, the secondary control unit destination address is non-zero and an ESCON
Director does not exist in the path. That is, the path is a direct connection.
11 Reserved.
12 Reserved.
13 / OK Fibre path established.
14 Fibre Channel Path Link Down.
15 Fibre Channel Path Retry Exceeded.
16 Fibre Channel Path Secondary Adapter not PPRC capable. This could be due to:
򐂰 Secondary Adapter not configured properly, or does not have the correct microcode loaded.
򐂰 The secondary adapter is already a target of 32 different ESS, DS8000, and DS6000.
17 Fibre Channel Path Secondary Adapter not available.
18 Fibre Channel Path Primary Login Exceeded.
19 Fibre Channel Path Secondary Login Exceeded.
Remote copy events
If you have configured Consistency Groups and a volume within this Consistency Group is
suspended due to a write error to the secondary device, trap 200, as shown in Example C-4,
will be sent out. One trap per LSS is configured with the Consistency Group option. This trap
could be handled by automation software like eRCMF to freeze this Consistency Group.
Example: C-4 Trap 200: LSS-pair Consistency Group PPRC-pair error

LSS-Pair Consistency Group PPRC-Pair Error
UNIT: Mnf Type-Mod SerialNm LS LD SR
PRI: IBM 2107-922 75-03461 56 84 08
SEC: IBM 2107-9A2 75-ABTV1 54 84
Trap 202, as shown in Example C-5, will be sent out if a remote Copy Pair goes into a
suspend state. The trap contains the serial number (SerialNm) of the primary and secondary
machine, the LSS (LS), and the logical device (LD). To avoid SNMP trap flooding, the number
of SNMP traps for the LSS will be throttled. The complete suspended pair information is
represented in the summary. The last row of the trap represents the suspend state for all pairs
in the reporting LSS. The suspended pair information contains a hexadecimal string of a
length of 64 characters. By converting this hex string into binary, each bit will represent a
single device. If the bit is 1, then the device is suspended; otherwise the device is still in full
duplex mode.
Example: C-5 Trap 202: Primary PPRC devices on LSS suspended due to error
Primary PPRC Devices on LSS Suspended Due to Error
UNIT: Mnf Type-Mod SerialNm LS LD SR
PRI: IBM 2107-922 75-20781 11 00 03
SEC: IBM 2107-9A2 75-ABTV1 21 00
Start: 2005/11/14 09:48:05 CST
PRI Dev Flags (1 bit/Dev, 1=Suspended):
C000000000000000000000000000000000000000000000000000000000000000
Global Mirror related SNMP traps

Trap 210, as shown in Example C-6, is sent out when a Consistency Group in a Global Mirror
environment is successfully formed.
Example: C-6 Trap 210: Asynchronous PPRC initial Consistency Group successfully formed
2005/11/14 15:30:55 CET
Asynchronous PPRC Initial Consistency Group Successfully Formed
UNIT: Mnf Type-Mod SerialNm
IBM 2107-922 75-20781
Session ID: 4002
Trap 211, as shown in Example C-7, will be sent out if the Global Mirror setup got into an
severe error state, where no attempts will be done to form a Consistency Group.
Example: C-7 Trap 211: Asynchronous PPRC session is in a fatal state

Asynchronous PPRC Session is in a Fatal State
IBM 2107-922 75-20781
Session ID: 4002

Trap 212, as shown in Example C-8, is sent out when a Consistency Group could not be
created in a Global Mirror Copy relation. Some of the reasons could be:
򐂰 Volumes have been taken out of a copy session.
򐂰 The remote copy link bandwidth might not be sufficient.
򐂰 The FC links between the PPRC primary and secondary system are not available.
Example: C-8 Trap 212: Asynchronous PPRC Consistency Group failure - Retry will be attempted
Asynchronous PPRC Consistency Group Failure - Retry will be attempted
IBM 2107-922 75-20781
Session ID: 4002
Trap 213, as shown in Example C-9, will be sent out when a Consistency Group in a Global
Mirror environment could be formed after a previous Consistency Group formation failure.
Example: C-9 Trap 213: Asynchronous PPRC Consistency Group successful recovery
Asynchronous PPRC Consistency Group Successful Recovery
IBM 2107-9A2 75-ABTV1
Session ID: 4002
Trap 214, as shown in Example C-10, will be sent out if a Global Mirror copy session is
terminated using the command rmgmir or the corresponding GUI function.
Example: C-10 Trap 214: Asynchronous PPRC master terminated

2005/11/14 15:30:14 CET
Asynchronous PPRC Master Terminated
IBM 2107-922 75-20781
Session ID: 4002
Trap 215, as shown in Example C-11, will be sent out if in the Global Mirror environment the
master has detected a failure to complete the FlashCopy commit. The trap will be sent out
after a number of commit retries have failed.
Example: C-11 Trap 215: Asynchronous PPRC FlashCopy at remote site unsuccessful
Asynchronous PPRC FlashCopy at Remote Site Unsuccessful
A UNIT: Mnf Type-Mod SerialNm
IBM 2107-9A2 75-ABTV1
Session ID: 4002
Trap 216, as shown in Example C-12, will be sent out if a Global Mirror master cannot
terminate the Global Copy relationship at one of his subordinates (slaves). This might occur if
the master is terminated with rmgmir but the master cannot terminate the copy relationship on
the subordinate. The user may need to run a rmgmir against the subordinate to prevent any
interference with other Global Mirror sessions.
Example: C-12 Trap 216: Asynchronous PPRC subordinate termination unsuccessful

Asynchronous PPRC Slave Termination Unsuccessful
Master: IBM 2107-922 75-20781
Slave: IBM 2107-921 75-03641
Session ID: 4002
Trap 217, as shown in Example C-13, will be sent out if a Global Mirror copy environment was
suspended by the DS CLI command pausegmir or the corresponding GUI function.
Example: C-13 Trap 217: Asynchronous PPRC paused

Asynchronous PPRC Paused
IBM 2107-9A2 75-ABTV1
Session ID: 4002

D
Appendix D. Licensing
In this appendix we describe how the licensing functions for Copy Services for the DS8000
Series are arranged.

Licenses
All DS8000 Series machines must have an Operating Environment License (OEL) for the
total storage installed, as defined in decimal TB. Licenses are also required for use of Copy
Services functions.
Licensed functions require the selection of a DS8000 series feature number (IBM 2107) and
the acquisition of DS8000 Series Function Authorization (IBM 2244) feature numbers:
򐂰 The 2107 licensed function indicator feature number enables technical activation of the
function subject to the client applying an activation code made available by IBM.
򐂰 The 2244 function authorization feature numbers establish the extent of IBM authorization
for that function on the 2107 machine for which it was acquired.
Table D-1 lists the DS8000 series feature numbers and corresponding DS8000 Series
Function Authorization feature numbers.
Table D-1 DS800 licensed functions

Licensed function IBM 2107 indicator feature IBM 2244 function
number authorization models and
feature numbers
Operating environment 0700 Model OEL 70xx
FICON Attachment 0702 Model OEL 7090
Point in time copy 0720 Model PTC 72xx
3-Site Metro/Global Mirror 0742 Model RMC 742x and 7430
Metro Mirror 0744 Model RMC 744x
Global Mirror 0746 Model RMC 746x and 7470
Remote Mirror for z/OS 0760 Model RMZ 76xx
Parallel access volumes 0780 Model PAV 78xx
Attention: For the DS8000 Turbo models 931, 932, and 9B2, the Metro Mirror and Global
Mirror features are now licensed separetely as indicated in Table D-1. For the former 92X
and 9A2 models, support for Metro Mirror and Global Mirror continues to be provided with
2244 model RMC features 740x and 7410, and indicator feature 0740.
The following is the breakdown of DS8000 Series Function Authorization feature numbers:
򐂰 OEL: Operating Environment License
– OEL - inactive 7020
– OEL - 1 TB 7021
– OEL - 5 TB 7022
– OEL - 10 TB 7023
– OEL - 25 TB 7024
– OEL - 50 TB 7025
– OEL - 100 TB 7030
򐂰 FICON: z/OS attachment
– FICON Attachment 7090
򐂰 PAV: Parallel Access Volumes
– PAV - inactive 7800
– PAV - 1 TB 7801
– PAV - 5 TB 7802
– PAV - 10 TB 7803
– PAV - 25 TB 7804
– PAV - 50 TB 7805
– PAV - 100 TB 7810
The following licenses apply to Copy Services:

򐂰 PTC: Point-in-Time Copy, also known as FlashCopy
– PTC - inactive 7200
– PTC - 1 TB 7201
– PTC - 5 TB 7202
– PTC - 10 TB 7203
– PTC - 25 TB 7204
– PTC - 50 TB 7205
– PTC - 100 TB 7210
򐂰 MGM: 3-Site Metro/Global Mirror
– MGM - inactive 7420
– MGM - 1 TB 7421
– MGM - 5 TB 7422
– MGM - 10 TB 7423
– MGM - 25 TB 7424
– MGM - 50 TB 7425
– MGM - 100 TB 7430
򐂰 MM: Metro Mirror
– MM - inactive 7440
– MM - 1 TB 7441
– MM - 5 TB 7442
– MM - 10 TB 7443
– MM - 25 TB 7444
– MM - 50 TB 7445
– MM - 100 TB 7450
򐂰 GM: Global Mirror
– GM - inactive 7460
– GM - 1 TB 7461
– GM - 5 TB 7462
– GM - 10 TB 7463
– GM - 25 TB 7464
– GM - 50 TB 7465
– GM - 100 TB 7470
򐂰 RMZ: Remote Mirror and Copy for z/OS, also known as z/OS Global Mirror (or XRC)
– RMZ - inactive 7600
– RMZ - 1 TB 7601
– RMZ - 5 TB 7602
– RMZ - 10 TB 7603
– RMZ - 25 TB 7604
– RMZ - 50 TB 7605
– RMZ - 100 TB 7610
Appendix D. Licensing 711

Additional information for Metro/Global Mirror licensing
When considering the 3-site Metro/Global Mirror solution, the following licensed functions are
required:
򐂰 For the DS8000 Turbo Models 931, 932, and 9B2:
– Site A - A Metro/Global Mirror (MGM) license and a Metro Mirror (MM) license
Additionally, a Global Mirror Add-on (GM Add) license is required if site B goes away
and you want to resync between site A and site C.
– Site B - A Metro/Global Mirror license (MGM), a Metro Mirror (MM) license, and a
Global Mirror Add-on (GM Add) license
– Site C - A Metro/Global Mirror license (MGM), a Global Mirror (GM) license, and a
Point-in-Time Copy (PTC) license
򐂰 For the D8000 Models 921, 922, and 9A2:
– Site A - A Metro/Global Mirror license (MGM) and a Remote Mirror and Copy (RMC)
license
– Site B - A Metro/Global Mirror license (MGM) and a Remote Mirror and Copy (RMC)
license
– Site C - A Metro/Global Mirror license (MGM), a Remote Mirror and Copy (RMC)
license, and a Point-in-Time Copy (PTC) license
DS Stotage Manager GUI support

The DS Storage Manager GUI panels used to define the licenced funtions and apply the
activation codes has been modified to reflect the separation of Metro Mirror and Global Mirror,
as well as the new Metro/Global Mirror, as illustrated in Figure 37-3, for the Apply Activation
codes panel.
Figure 37-3 Apply Activation Codes panel
Authorized level
All Copy Services functions require licensing to be activated. This means that the customer
must purchase a license for the appropriate level of storage for each Copy Service function
that is required. They then have to install the License Key generated using the Disk Storage
Feature Activation application (DSFA). Another consideration relates to the authorized level
required. In most cases the total capacity installed must be licensed. This is the total capacity
in decimal TB equal to or greater than the actual capacity installed, including all RAID parity
disks and hot spares.
An exception may be where a mix of both System z and open systems hosts are using the
same storage server. In this case it is possible to acquire Copy Services licenses for just the
capacity formatted for CKD, or just the capacity formatted for FB storage. This implies that the
licensed Copy Services function is needed only for open systems hosts, or only for System z
hosts. If, however, a Copy Services function is required for both CKD and FB, then that Copy
Services license needs to match the total configured capacity of the machine. The
authorization level is maintained by the licensed code in the controller and the DSFA
application.
For example, the actual capacity is 15 TB, used for both CKD and FB, the scope for the OEL
is therefore a type of ALL and the installed OEL must be at least 15 TB. If the client has split
storage allocation, with 8 TB for CKD, and only CKD storage is using FlashCopy, then the
scope type for the PTC license can be set to CKD. Now the PTC license can be purchased at
the CKD level of 8 TB. However, this means that no open systems hosts can use the
FlashCopy function.
The actual ordered level of any Copy Service license can be any level above that required or
installed. Licenses can be added and have their capacities increased, non-disruptively to an
installed system.
Important: A decrease in the scope or the capacity of a license requires a disruptive IML
of the DS8000 Storage Facility Image.
Charging example
A client can choose to purchase any or all DS8000 licenses at an authorization level at or
later than the installed raw disk capacity. It may be more cost effective to pre-install
authorization to use greater than the currently installed storage capacity.
A simple rule for charges is remembering that the required authorization level depends upon
the license scope:
򐂰 If the license scope is FB, the authorization level must be equal to or greater than the total
amount of physical capacity within the system that will be logically configured as FB.
򐂰 If the license scope is CKD, the authorization level must be equal to or greater than the
total amount of physical capacity within the system that will be logically configured as
CKD.
򐂰 If the license scope is ALL, the authorization level must be equal to or greater than the
total system physical capacity.
The dual Storage Facility Image (SFI) version of DS8300 (the model 9Bx) uses the same
logic, in this case added between the two SFIs. So all of the above apply to each SFI
separately. It is possible to license Copy Services for a single scope in one SFI.
Appendix D. Licensing 713

Here is an explanation of this scenario:
򐂰 If you activate a licensed function on only SFI 1 with a license scope of FB:
The authorization level must be equal to or greater than the total amount of physical
capacity within SFI 1 that will be logically configured as FB.
򐂰 If you activate a licensed function on SFI 1 and SFI 2, both with a license scope of CKD:
capacity within both SFI 1 and SFI 2 that will be logically configured as CKD.
򐂰 If you activate a licensed function on SFI 1 with a license scope of FB and on SFI 2 with a
license scope of ALL:
capacity within SFI 1 that will be logically configured as FB and the total physical capacity
within SFI 2.
E
Appendix E. CLI migration

This chapter discusses a proposed way of how you can migrate Copy Services tasks for the
ESS environment to the DS Copy Services environment. The Copy Services functions
described here cover the Graphical User Interface (GUI) and the command-line interface
(CLI).

Migrating ESS CLI to DS CLI
With the introduction of the IBM DS8000 Storage Unit, a new Copy Services application is
also introduced. The Copy Services functions can be issued via the DS graphical user
interface (GUI) or via the DS CLI.
The Advanced Copy Services functions that are available on the ESS 800 are also available
on the DS8000. Although the functions are still available, there are also some differences that
need to be considered in replacing your ESS CLI with a DS CLI. These are:
򐂰 Point-in-Time Copy (FlashCopy) does not support Consistency Groups on the GUI.
򐂰 Fibre Channel is used for Metro Mirror, Global Mirror, and Metro/Global Copy.
򐂰 The GUI runs real-time only (tasks cannot be saved), while the CLI can be invoked using a
saved script.
򐂰 The DS CLI supports both the ESS 800/750 and the DS8000.
򐂰 ESS 800 must be at Licensed Internal Code (LIC) level 2.4.3.15 or later.
Review the ESS tasks to migrate

Review what Copy Services tasks you wish to migrate. You may check these tasks from your
ESS GUI or ESS CLI. Example E-1 shows the esscli list command to display the tasks.
Example: E-1 esscli list task

esscli list task –s copy_services-server –u csadmin –p passw0rd

Task Name Type Status
------------------------------------------------------------------------
H_Epath_test16 PPRCEstablishPaths NotRunning
H_Epath_test17 PPRCEstablishPaths NotRunning
Brocade_pr_lss10 PPRCEstablishPair NotRunning
Brocade_pr_lss11 PPRCEstablishPair NotRunning
Flash10041005 FCEstablish NotRunning
You can use ESS CLI to display the contents of each saved task and write the contents to a
file. See Example E-2.
Example: E-2 esscli show task

esscli show task –s copy_services_server –u csadmin –p passw0rd –d “name=Flash10041005”

Taskname=Flash10041005
Tasktype=FCEstablish
Options=NoBackgroundCopy
SourceServer=2105.23953
TargetServer=2105.23953
SourceVol TargetVol
------------------------------------------------------------------------
1004 1005
You can also check your saved tasks via the ESS GUI. See Figure E-1.
Figure E-1 ESS Copy Services GUI tasks panel
Highlight the task and click the information panel. See Figure E-2.
Figure E-2 ESS task information
Also review the specific server scripts (depending on the OS) that perform tasks that set up
and execute saved ESS CLI saved tasks. You may need to edit or translate these scripts in
order to run your DS CLI saved tasks.
Important: On the ESS 800, open systems volume IDs are given in an 8-digit format,
xxx-sssss, where xxx is the LUN ID and sssss is the serial number of the ESS 800. In the
example used in this appendix the volumes shown are 004-23953 to 005-23953. These
volumes are open systems or fixed block volumes. When referring to them in the DS CLI,
you must add 1000 to the volume ID, so volume 004-23953 is volume ID 1004 and volume
005-23953 is volume ID 1005. This is very important because on the ESS 800, the
following address ranges are actually used:
0000 to 0FFF System z CKD volumes (4096 possible addresses)
1000 to 1FFF Open systems fixed block LUNs (4096 possible addresses)
If we intend to use FlashCopy to copy ESS LUN 004-23953 onto 005-23953 using the DS
CLI, we must specify 1004 and 1005. If instead we specify 0004 and 0005, we will actually
run the FlashCopy against CKD volumes. This may result in an unplanned outage on the
System z host that was using CKD volume 0005.
The ESS CLI command, show task, will show the correct value for the volume ID.
Appendix E. CLI migration 717

Convert the individual tasks
Choose the ESS CLI tasks that you need to translate to the DS CLI. Refer to the IBM System
Storage DS8000 Command-Line Interface User´s Guide, SC26-7916. You can then save
each translated task and be able to run it in the DS8000 CLI environment.
Note: The DS GUI (via the DS HMC) does not save Copy Services tasks like the ESS GUI.
You can only use the DS GUI for Copy Services in real-time mode.
Also translate (if needed) and edit the individual server scripts that will set up and execute the
saved DS CLI scripts. See Table E-1 for an example of command translation.
Table E-1 Converting ESS CLI to DS CLI

Task parameter ESS CLI parameter DS CLI conversion Description
Tasktype FCEstablish mkflash Establish FlashCopy.
Options NoBackgroundCopy -nocp No background copy

upon FlashCopy.
SourceServer 2105.23953 -dev IBM.2105-23953 Device ID.
TargetServer 2105.23953 N/A Used only once in DS

CLI.
Source and Target 1004 1005 1004:1005 Separated by a colon

vols. in DS CLI.
DS CLI commands
Example E-3 is the translation to DS CLI commands.
Example: E-3 DS CLI mkflash command

dscli> mkflash -nocp -dev IBM.2105-23953 1004:1005
dscli> lsflash -dev IBM.2105-23953 1004:1005
ID SrcLSS SequenceNum Timeout ActiveCopy Recoding Persistent Revertible
===============================================================================
See Figure E-3 for the GUI output.
Figure E-3 ESS GUI FlashCopy window
ESS/DS CLI comparison
Table E-2 shows a brief comparison of the major components between the ESS CLI and the
DS CLI.
Table E-2 ESS and DS CLI commands and parameters comparison

ESS CLI DS CLI Comments
list server lsserver Like the 2105, the 2107 storage

facility image contains one pair
of servers.
list volumespace lsextpool, showextpool, See Note 1.

lsrank, showrank, lsarray,
showarray, lsarraysite
create volumespace mkextpool, mkarray, mkrank
delete volumespace rmrank, rmarray, rmextpool
list diskgroup lsarraysite A 2107 Array Site consists of

eight disk drives (DDMs) that
are made into a RAID array.
The 2107 does not support the
JBOD Array configuration.
list port lsioport, showioport The 2107 CLI lsioport and

showioport commands include
the –metrics parameter, which
returns the performance
counter values for the
respective I/O port IDs. The
–metrics parameter provides
the means to monitor I/O port
performance statistics.
set port setioport See Note 2.
list volume lsfbvol, lsckdvol See Note 3.
create volume mkfbvol, mkckdvol
set volume chfbvol, chckdvol
list pav lsckdvol, showckdvol
create pav mkckdvol
delete pav rmckdvol
list volumeaccess lsvolgrp, showvolgrp See Note 4.
create volumeaccess mkvolgrp, chvolgrp
delete volumeaccess rmvolgrp
list hostconnection lshostconnect, The 2107 CLI commands

showhostconnect include the volume group ID
parameter.
create hostconnection mkhostconnect For 2107, the hostconnect
commands concern SCSI-FCP
delete hostconnection rmhostconnect
host port connections to ESS
set hostconnection chhostconnect I/O ports that are configured for
SCSI-FCP and identified
access mode.

ESS CLI DS CLI Comments
list log N/A
list featurecode lsda, lshba, lsioencl, lsuser, The 2107 CLI commands can
mkuser, rmuser, chuser, display feature codes when the
lsstgencl appropriate parameters are
used with the commands.
list task NA
show task NA
list pprcpaths lspprcpath Unlike the 2105, the 2107 CLI

Copy Services functions are not
task oriented. The 2107 CLI
provides a complete set of
FlashCopy and PPRC make,
change, remove, list, and show
commands.
rsExecuteTask mkflash, rmflash, mkpprc, The 2107 CLI provides a

rmpprc, mkpprcpath, complete set of FlashCopy and
rmpprcpath, mksession, PPRC commands that may be
chsession, rmsession used in the coding of scripts
that emulate 2105 Copy
Services tasks.
rsList2105s lssu The 2107 CLI lssu command

returns both 2105 and 2107
objects that are contained by
the ESS Network Interface
domain.
rsQuery, rsQueryComplete, lsflash, lspprc These 2107 Copy Services CLI

rsFlashCopyQuery commands are equivalent to
the respective 2105 CLI
commands.
The 2107 mkflashcopy and
mkpprc commands provide a
wait flag that delays command
response until copy complete
status is achieved.
rsTestConnection lsaavailpprcport
Note 1: Volume space configuration is a primary difference between 2105 and 2107. Like
the 2105, a 2107 Storage Facility Image volume space contains a RAID 5 or RAID 10 Array
and Rank that are configured from an Array Site. For the 2105, one command configures
an Array Site into a RAID array and Rank. For the 2107, one command configures an Array
Site into an Array, and a second command configures an Array into a Rank. For 2105, a
Rank is configured as fixed block or CKD. For the 2107, a Rank is assigned to a
user-defined Extent Pool object, which the user defines as either the fixed block or CKD
storage type. The interleave volume construct does not exist for 2107. For the 2105, a
volume is configured from a specific Rank, and cannot span Rank boundaries. For the
2107, a volume is configured from an Extent Pool. An Extent Pool can contain multiple
Ranks. A 2107 volume consists of one or more Extents that can be allocated from one or
more Ranks. A fixed block Extent is 1 GB (128 logical blocks). Each block contains 512
bytes of usable data space.
For 2105, a Rank is either assigned to server 0 or server 1, depending on the Array Site
location. A 2105 Rank is assigned to one of 32 possible LSS IDs, depending on the device
adapter pair location and storage type configuration.
For 2107, an Extent Pool is assigned to server 0 or server 1. A Rank that is configured from
any Array Site can be assigned to a server 0 or 1 Extent Pool. Array Site position and
device adapter pairs are not factors for the Rank to Extent Pool assignment. A volume that
is created from a server 0 Extent Pool is assigned to an even-numbered LSS ID. A volume
created from a server 1 Extent Pool is assigned to an odd numbered LSS ID. A user must
define at least two Extent Pools (0 and 1), but can define as many Extent Pools as there
are Ranks. For 2105, a user can delete a Rank but cannot delete a volume. For 2107, a
user can delete a single volume, Rank, or Extent Pool. The 2107 CLI showrank and
showextpool commands include a metrics parameter that returns the performance counter
values for a specified Rank or Extent Pool ID. The metrics parameter provides the means
to monitor Rank and Extent Pool performance statistics.
Note 2: Like 2105, a 2107 Fibre Channel SCSI-FCP I/O port can be configured for either
the point-to-point/switched fabric or FC-AL connection topologies. A port that uses the
point-to-point/switched fabric topology can be simultaneously used for OS host system I/O
and for PPRC path configurations. The Fibre Channel SCSI-FCP IO port only allows
identified host system ports to access volumes. A host system port WWPN must be
identified (registered) to each ESS IO port through which volume access is intended. Host
system port WWPN identification is accomplished by the CLI mkhostconnect command.
Note 3: A 2107 storage facility image can contain up to 65,536 volumes. A 2105 Storage
Unit can contain up to 4096 FB volumes and 4096 CKD volumes. Otherwise, the 2105 and
2107 volume definitions and characteristics are essentially identical.
The 2107 CLI provides a specific set of volume commands for each storage type, fixed
block or CKD, as a means to clarify input parameter and output device adapter definitions.
The 2107 CLI showfbvol and showckdvol commands include a metrics parameter that
returns the performance counter values for a specified volume ID. The metrics parameter
provides the means to monitor volume performance statistics.

Note 4: The 2105 volumeaccess commands concern volume ID assignment to a
SCSI-FCP host port initiator, or WWPN. For 2107, volume IDs are assigned to a
user-defined volume group ID (mkvolgrp and chvolgrp). A volume group ID is then
assigned to one or more host system ports (mkhostconnect and chhostconnect) as a
means to complete the volume access configuration.
The volume group construct also exists in the 2105 internal code, but the construct is not
externalized by the 2105 Specialist or CLI commands.
For 2107 fixed block volumes, a Volume Group must be configured as either SCSI-mask or
SCSI-map-256, depending whether the volume group is accessed by a SCSI-FCP host
port that used the report LUNs or poll LUNs access method protocol.
Related publications
The publications listed in this section are considered particularly suitable for a more detailed
discussion of the topics covered in this book.
IBM Redbooks
For information about ordering these publications, see “How to get IBM Redbooks” on
page 725. Note that some of the documents referenced here may be available in softcopy
only.
򐂰 The IBM TotalStorage DS8000 Series: Implementation, SG24-6786
򐂰 IBM TotalStorage Business Continuity Solutions Guide, SG24-6547
򐂰 IBM System Storage Solutions Handbook, SG24-5250
򐂰 The IBM TotalStorage DS8000 Series: Copy Services with IBM Eserver zSeries,
SG24-6787
If you are implementing Copy Services in a mixed technology environment you may be
interested in referring to the following manuals on the ESS and DS6000.
򐂰 IBM TotalStorage Enterprise Storage Server Implementing ESS Copy Services in Open
Environments, SG24-5757
򐂰 IBM TotalStorage Enterprise Storage Server Implementing ESS Copy Services with
IBM Eserver zSeries, SG24-5680
򐂰 The IBM TotalStorage DS6000 Series: Copy Services with IBM Eserver zSeries,
SG24-6782
򐂰 The IBM TotalStorage DS6000 Series: Copy Services in Open Environments Redbook,
SG24-6783
򐂰 DFSMShsm ABARS and Mainstar Solutions, SG24-5089
򐂰 Practical Guide for SAN with pSeries, SG24-6050
򐂰 Fault Tolerant Storage Multipathing and Clustering Solutions for Open Systems for the IBM
ESS, SG24-6295
򐂰 Implementing Linux with IBM Disk Storage, SG24-6261
򐂰 Linux with zSeries and ESS: Essentials, SG24-7025
Other publications
These publications are also relevant as further information sources. Note that some of the
documents referenced here may be available in softcopy only.
򐂰 IBM System Storage DS8000 Command-Line Interface User´s Guide, SC26-7916
򐂰 IBM System Storage DS8000: Host Systems Attachment Guide, SC26-7917
򐂰 IBM System Storage DS8000: Introduction and Planning Guide, GC35-0515
򐂰 IBM System Storage Multipath Subsystem Device Driver User’s Guide, SC30-4131
򐂰 IBM System Storage DS8000: User’s Guide, SC26-7915

򐂰 IBM System Storage DS Open Application Programming Interface Reference, GC35-0516
򐂰 IBM System Storage DS8000 Messages Reference, GC26-7914
򐂰 z/OS DFSMS Advanced Copy Services, SC35-0248
򐂰 Device Support Facilities: User’s Guide and Reference, GC35-0033
Online resources
These Web sites and URLs are also relevant as further information sources:
򐂰 IBM Disk Storage Feature Activation (DSFA) Web site:
򐂰 The PSP information:
http://www-1.ibm.com/servers/resourcelink/svc03100.nsf?OpenDatabase
򐂰 Documentation for the DS8000:
http://www.ibm.com/servers/storage/support/disk/2107.html
򐂰 The Interoperability Matrix:
http://www.ibm.com/servers/storage/disk/ds8000/interop.html
򐂰 Fibre Channel host bus adapter firmware and driver level matrix:
http://knowledge.storage.ibm.com/servers/storage/support/hbasearch/interop/hbaSearch.do
򐂰 ATTO:
http://www.attotech.com/
򐂰 Emulex:
http://www.emulex.com/ts/dds.html
򐂰 JNI:
http://www.jni.com/OEM/oem.cfm?ID=4
򐂰 QLogic:
http://www.qlogic.com/support/ibm_page.html
򐂰 IBM:
http://www.ibm.com/storage/ibmsan/products/sanfabric.html
򐂰 McDATA:
http://www.mcdata.com/ibm/
򐂰 Cisco:
http://www.cisco.com/go/ibm/storage
򐂰 CIENA:
http://www.ciena.com/products/transport/shorthaul/cn2000/index.asp
򐂰 Nortel:
http://www.nortelnetworks.com/
򐂰 ADVA:
http://www.advaoptical.com/
How to get IBM Redbooks
You can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft
publications and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at
this Web site:
ibm.com/redbooks
Help from IBM

IBM Support and downloads
ibm.com/support
IBM Global Services

ibm.com/services
Related publications 725

Index
remote FlashCopy 78
A structure 26
activating FlashCopy 58 commitflash 72, 333, 401
Active/Active 587 configurations
Active/Passive 587 activating FlashCopy 58
add paths to existing paths between LLSs 164 planning for FlashCopy 58
AIX 596 connectivity
FlashCopy 592 for Global Mirror to remote site 254
specifics 592 Consistency Group 118, 259, 383, 401, 578
architecture data consistency 117
Copy Services 7 formation 260
eRCMF 580 parameters 261
asynchronous data replication 248 verify state 277
Asynchronous PPRC see Global Mirror what is it 116
attributes Consistency Group (CG) 380
consistent asynchronous remote copy solution 251 Consistency Group drain time 368
synchronous data replication 248 Consistency Group FlashCopy 48
authorized level Consistency Group interval time 368, 373
licensing 713 consistencygrp 398
automation and management 112 consistgrp 387
Continuous Availability (CA) 630
B Continuous Operations (CO) 630
B volumes coordinate 312
consistent data 280 Copy Pending 397
bandwidth Copy Services 4, 613
Metro Mirror 124 1750 DS6000 12
basic concepts 2105 ESS 800 12
FlashCopy 39 architecture 7
basic concepts of Global Mirror 252 differences of DS CLI and DS GUI
DS CLI commands 27
DS8000 11
C DS8000 and ESS interoperability 512
capacity establishment errors 503
Global Copy 212 how the new structure of Copy Services works 11
cascade 388 HP-UX 613
cascading 491 introduction to new structure 8
cfgmgr 596 network components 135
cginterval 312 Remote Mirror and Copy between Storage Complex-
charging example 713 es 13
chhostconnect 722 using VERITAS Volume Manager
chkpprc 641 SUN Solaris 610
chlss 398 what is a Storage Complex 10
chsession 290, 317, 416 Windows volumes 599
chvolgrp 722 Copy Services Domain
CLI migration adding 514
convert the individual tasks 718 Copy Services network components
migrating ESS CLI to DS CLI 716 Metro Mirror 135
review ESS tasks to migrate 716 create
cluster aware 588 Global Copy relationship between local and remote
clustering 115 volume 255
command-line configuration Global Mirror environment 265
examples 137 creating DS HMC on another PC 21
commands creating paths
comparison for Global Copy 202 Metro Mirror 160
FlashCopy 67 Cross Site Mirroring 633

D user assistance 31
data backup system 38 DS Command-Line Interface see DS CLI
data consistency 111, 117–118, 492 DS front ends
Consistency Group 117 FlashCopy commands 64
data migration 492 DS GUI
data mining system 38 accessing Software Information Center 21
decoding port IDs 520 advantages over DS CLI 18
default profile 26 determine DS6000 WWNN 504
define determine DS8000 WWNN 505
Global Mirror session 257 disadvantages over DS CLI 19
delete exisiting FlashCopy relationship using the DS SM establish paths 230
89 Global Copy examples 230
delete paths 169 Global Mirror 291
dependent writes 244, 248, 577 Global Mirror examples 355
Disaster Recovery manage Global Copy pairs 506
practice readiness 347 manage Global Mirror environment 369
Disaster Recovery test 429 managing Metro Mirror pairs 506
disaster recovery test scenarios 427 Metro Mirror 136
distance path creation 520
Metro Mirror 125 DS SM 64, 80–82, 84–85, 89, 288
double cascading 491–492 accessing when logged onto HMC 18
drain 312 delete existing FlashCopy relationship 89
draining 401 establish Global Mirror environment 355
DS API 64 resynchronize target 88
DS CLI 13, 64, 78, 81, 84, 288, 716 DS6000
advantages over DS GUI 19 adding DS8000 Storage Complex 499
application 29 Copy Services 12
command structure 26 create remote FlashCopy using DS CLI 509
Copy Services commands 27 determine WWNN using DS GUI 504
create DS6000 remote FlashCopy 509 how to determine WWNN using DS CLI 504
default profile 26 interoperabilitly of Copy Services with DS8000 495
determine DS6000 WWNN 504 SFI 9
determine DS8000 WWNN 505 DS8000
determine WWNN for ESS 800 524 adding DS6000 Storage Complex 498
disadvantages over DS GUI 18 Copy Services 4, 11
establishing logical paths 504, 525 Copy Services with ESS 512
establishing the logical paths 505 create a user ID 513
functionality 24 create DS HMC on another PC 21
Global Copy examples 214 data consistency 111
Global Mirror 289 determine WWNN using DS CLI 505
Global Mirror examples 306 determine WWNN using DS GUI 505
help 31 DS CLI highlights 24
highlights 24 FlashCopy 4
interactive mode 29–30 interoperability of Copy Services with DS6000 495
introduction 24 interoperability with ESS 800 511
listing available ports 505 machine signature 61
local FlashCopies 66 procedure to add Storage Complex 19
manage Global Copy pairs 507 Remote Mirror and Copy 5
manage Global Mirror pairs 508 SFI 9
manage Metro Mirror pairs 506 dspessdta 641, 644
Metro Mirror 136
operating systems 24 E
profile 25 Enterprise level
return codes 31 eRCMF 582
script command mode 29 enterprise Remote Copy Management Facility see eRC-
script mode 29 MF
single-shot mode 29 environment
updating profile 513 remove Global Mirror 270
updating the profile 497 eRCMF 134, 288, 293, 575, 579
user accounts 25 architecture 580
Enterprise level 582 Fibre Channel
Freeze and Go 583 Metro Mirror links 122
Freeze and Stop 583 FlashCopy 4, 75, 613
Master Process 582 activating 58
solution highlights 576 and Global Copy 44
Tier 4 580 and Global Mirror for open systems 45
Tier 6 580 and Metro Mirror 43
VolumeSet level 582 apply changes to existing FlashCopy relationship 85
ESS basic concepts 39
Copy Services with DS8000 512 between virtual machines 621
determining volume size 516 between VMware ESX servers 623
ESS 800 commands and parameters used 67
Copy Services 12 commands in the DS front ends 64
create a user ID 513 commit data to target using commitflash 72
interoperability with DS8000 511 comparison of display properties from the DS CLI and
volume address considerations 519 DS SM 84
ESS CLI 716 Consistency Group FlashCopy 48
ESS tasks to migrate 716 data backup system 38
establish a Global Mirror environment with DS SM 355 data mining system 38
establish pair delete existing FlashCopy relationship using the DS
Global Copy 192 SM 89
example display existing FlashCopies using lsflash 70
failover/failback 286 display properties of existing FlashCopy using the DS
examples SM 82
add paths to existing paths between LLSs 164 establish 39
command-line configuration 137 existing Global Copy primary 49
creating Metro Mirror pairs 171 existing Metro Mirror 49
creating paths 160 fast reverse restore 54
delete paths 169 flow 64
DS Storage Manager GUI 230 Freeze FlashCopy Consistency Group 49
establish a Global Mirror environment with DS SM full volume copy 42
355 HP-UX 613
establish Global Copy pairs with DS GUI 234 increment FlashCopy using resyncflash 73
establish paths with the DS Storage Manager GUI Incremental FlashCopy 50
230 initiate background copy for persistent FlashCopy re-
failback 181 lationship 87
failover 179 initiate using Create 80
FlashCopy 102 initiate using mkflash 69
Global Copy and DS CLI 214 integration system 39
Global Mirror and DS CLI 306 interfaces 55, 63
manage Global Mirror environment with DS GUI 369 introduce 256
modify a Global Mirror session 373 limitations with multiple FlashCopies 48
pause a Global Mirror session 372 local 65
resume 177 local commands 68
resume a Global Mirror session 372 local FlashCopies using the DS CLI 66
simplifying Metro Mirror commands 137 managing for ESS 800 532
suspend 175 multiple relationship 48
exportvg 596 nocopy option 42
Extended Long Busy (ELB) 578 operational areas 38
extlongbusy timeout 398 options 47, 55
overview 37
parameters for initial FC using DS SM and DS CLI 81
F parameters used in remote FlashCopy 78
failback 115, 181 performance 92
Metro Mirror 181 Persistent 53
failover 115, 396, 408 planning 58
B to A 276 production backup system 38
Metro Mirror 179 reading from the source 41
failover/failback example 286 reading from the target 41
failoverpprc 331, 343, 409 remote 52, 65
fast reverse restore 54
Index 729
remote FlashCopies using the DS CLI 78 create PPRC paths 215
remote FlashCopy commands 78 determine available fibre links 215
remove local FlashCopy using rmflash 77 establish pair 192
remove relationship 269 establish pairs with DS GUI 234
reset FlashCopy Consistency Group using unfreeze- examples 214
flash 78 examples with DS GUI 230
reset target to contents using revertflash 76 monitoring the copy status with DS GUI 238
resynchronize target using the DS SM 88 overview 188, 202
reverse existing FlashCopy using the DS SM 85 pausepprc 220
reverse restore 54 performance 212
run new background copy for persistent FlashCopy positioning 190
using rmflash 77 primary 49
set an existing FlashCopy to revertible using set- remove pairs 217
flashrevertible 72 remove PPRC paths 218
terminating the FlashCopy relationship 41 resume 220
test system 39 resumepprc 221
using the DS SM front end 80 rmpprc 217
VMware ESX server 617 scalability 212
with other Copy Services 43 setting up environment 215
within a virtual machine 619 state change logic 189
writing to the source 41 suspend 220
writing to the target 41 symmetrical configuration 199
FlashCopy examples Global Mirror 4, 6, 248, 370
create backup 103 add A volumes to session on each LSS 310
create test system or integration system 102 add and remove A volume to existing environment
creating a FlashCopy for backup purposes without 325
volume copy 103 add and remove LSS to existing Global Mirror environ-
multiple setup of a test system with same contents ment 327
102 add and remove Subordinate 329
one time test system 102 add or remove storage servers or LSSs 268
using a target volume to restore its contents back to attributes of synchronous data replication 248
the source 104 basic concepts 252
using an Incremental FlashCopy for backup purposes change tuning parameters 323
104 clear up environment with DS CLI 315
flow close session 318
FlashCopy 64 connectivity to remote site 254
formation Consistency Group drain time 299
Consistency Group 260 Consistency Group formation 260
freeze 120, 579 Consistency Group parameters 261
Freeze and Go consistent asynchronous remote copy solution 251
eRCMF 583 consistent data on B volumes 280
Freeze and Stop coordination time 298
eRCMF 583 create 265
Freeze FlashCopy Consistency Group 49 create FlashCopy with DS GUI 363
freezepprc 393, 398, 429 create Global Copy with DS GUI 359
Full Duplex 396 create PPRC paths from A to B 342
full volume copy 42 create PPRC paths from B to A 339
create session with DS GUI 367
define session 257
G dependent writes 244, 248
GDPS 579 DS CLI 289
GDS 585 DS GUI 291
Global Copy 4–5 DS GUI examples 355
adding and removing PPRC paths 224 eRCMF 293
capacity 212 establish environment with DS GUI 355
changing mode to Metro Mirror 221, 223 failback Global Copy from A to B 344, 352
clear environment 217 failback Global Copy from B to A 339
command comparison 202 failover 276
consistent point-in-time copy 193 failover from B to A 331
converting to Metro Mirror with DS GUI 239 failover Global Copy from A to B 343
create pairs 215
failover Global Copy from B to A 349 verify for Consistency Group state 277
failover/failback example 286 verify for valid Consistency Group state 332
for open systems 45 view session’s volumes 371
form Consistency Group 259 volumes 267
interfaces 288 wait for Global Copy first pass to complete 354
introduce FlashCopy 256 Global Mirror Master 383
local site 284 Global Mirror session 253
manage environment 320 pause 372
modify parameters 268
modify session 267, 373
multiple disk storage servers 270 H
non-revertible 278 heartbeat 634
operation 275 help 32
pause 348 DS CLI 31
pause and resume Consistency Group formation 320 Software Information Center 21
pause Global Copy pairs from A to B 349 High Availability (HA) 630
perform disaster recovery testing on D volume 352 HP-UX 613, 615
performance considerations at coordination time 298 hypervisor 631
populate session with volumes 258
primary site failure 275 I
query for Global Copy first pass completion 341 I/O port 357
query out-of-sync tracks for Global Copy 342 IBM TotalStorage Continuous Availability for Windows
quiesce application at remote site 342 585–586
recovery after local site failure using DS CLI 330 introduction 586
Recovery Point Objective (RPO) 296 overview 586, 588
recovery scenario 274 service and ordering 589
reestablish FlashCopy from B to C 337 solution highlights 586
reestablish FlashCopy relationship 350 IBM TotalStorage Rapid Data Recovery 575
reestablish FlashCopy relationship between B and C IBM VDS hardware provider 606
volumes 282 IBM VSS Provider 602
relationship between local and remote volume 255 importvg 596
remote storage server configuration 299 Incremental FlashCopy 50
remove A volumes from session 317 Incremental Resync 437
remove environment 270 Independent Auxiliary Storage Pool (IASP) 635
remove FlashCopy between B and C volumes 319 individual tasks
remove FlashCopy relationship 269 CLI migration 718
remove Global Copy pairs 319 initial disk synchronization
remove PPRC paths 319 Metro Mirror 114
replicating data 244 initial synchronization 130
restart application 282 initiate FlashCopy using Create 80
restart application at remote site 338 Input output processor (IOP) 632
resume 354 integration system 39
return to local site 338 interactive mode
reverse FlashCopy from B to C 334, 350 DS CLI 29–30
revert action 280 interfaces
revertible 278 FlashCopy 55, 63
session 253, 259 Global Mirror 288
simple configuration 254 remote FlashCopy 65
start 345 interoperability 489
start application at the local site 347 DS6000 and DS8000
start for a specified session 311 adding the Storage Complex 498
start session on each LSS 309 creating user IDs and passwords 497
start with Subordinate 314 establish logical paths using DS CLI 504–505
summary of recovery scenario 330 establishing RMC paths 503
synchronous data replication 244 FlashCopy 509
take FlashCopy from B to D 351 hardware requirements 496
terminate 316, 330 licensing requirements 496
terminate and start 324 list available ports using DS CLI 505
terminology 264 manage Global Copy pairs using DS CLI 507
topology 269 manage Global Copy pairs using DS GUI 506
Index 731
manage Global Mirror pairs using DS CLI 508 Master disk subsystem 380
manage Metro Mirror pairs using DS GUI 506 Master Process
manage Metro Mirror pairs using the DS CLI 506 eRCMF 582
microcode levels 496 maximum coordination interval 368, 373
network connectivity 496 maximum time writes inhibited to remote site 373
path creating using the DS GUI 504 metrics 391
storage management 501 Metro Mirror 4–5, 49
updating the DS CLI profile 497 adding capacity in new DS6000s 131
user ID management 500 automation and management 112
volume size considerations for RMC 501 bandwidth 124
DS6000 and DS8000 Copy Services 495 changing options 168
DS6000 and ESS 800 clustering 115
managing ESS 800 FlashCopy 532 Copy Services network components 135
managing Metro Mirror or Global Copy pairs 526 creating pairs 171
DS8000 and ESS 496 creating paths 160
DS8000 and ESS Copy Services 512 data consistency 111
ESS 800 and DS8000 511 distance 125
adding Copy Services Domain 514 DS CLI 136
Copy Services 512 DS GUI 136
volume size considerations for RMC 515 examples 137
introduce FlashCopy 256 failback 115, 181
introduction failover 115, 179
Copy Services 4 failover and failback 115
IBM TotalStorage Continuous Availability for Windows Fibre Channel links 122
586 freeze 120
RMC 5 initial disk synchronization 114
iSeries Access for Windows 643 initial synchronization 130
iSeries Navigator 643 links 121
logical paths 123
LSS design 124
L overview 110
licensing 710 pausepprc 144
authorized level 713 performance 130
charging example 713 resume example 177
links resumepprc 145
Metro Mirror 121 rmpprc 142
local 283 rolling disaster 112
FlashCopy 65 scalability 130
local FlashCopy commands symmetrical configuration 125
parameters 67 traps 705
local site volumes 126
return 284 Metro/Global Mirror 6, 378
logical paths MGM 378
Metro Mirror 123 configuration 382
long busy 398 multiple subsystems 383
lsflash 68, 70, 73, 402 planned recovery 407
lspprc 397 Microsoft Virtual Disk Service see VDS
LSS 268 Microsoft Volume Shadow Copy Services see VSS
add or remove in Global Mirror 268 mkflash 32, 69–70
LSS design mkgmir 290, 346, 390
Metro Mirror 124 mkhostconnect 722
lssession 290, 346, 392 mkpprc 215
LVM 597 mkpprcpath 387
mkremoteflash 229
M mksession 290, 389
man page 32 mkvolgrp 722
manage Global Mirror environment with DS GUI 369 modify a Global Mirror session 267
management console 8 modify a session
managepwfile 26, 498, 514 Global Mirror 373
Master 400 modify Global Mirror session parameters 268
multiple relationship Point-in-time Copy see PTC
FlashCopy 48 populate Global Mirror session 258
port IDs
decoding 503, 520
N positioning
nocopy option 42 Global Copy 190
nocp 389 PPRC
non-revertible 278 volume size considerations 515
PPRC links 386
O PPRC path
open systems reason codes 704
AIX and FlashCopy 592 PPRC paths 386
AIX and Remote Mirror and Copy 596 PPRC-XD see Global Copy
AIX specifics 592 primary site failure 275
Copy Services using VERITAS Volume Manager, procedure
SUN Solaris 610 periodic offsite backup 224
Copy Services with Windows volumes 599 production backup system 38
HP-UX and Copy Services 613 profile
HP-UX and FlashCopy 613 DS CLI 25
HP-UX with Remote Mirror and Copy 615 PTC 4
Microsoft Virtual Disk Service (VDS) 604
Microsoft Volume Shadow Copy Services (VSS) 601 Q
SUN Solaris and Copy Services 609 queue full 429
Windows and Remote Mirror and Copy 598 Queue Full condition 578
open systems specifics
Virtual Disk Service overview 604
operational areas 38 R
options reason codes 704
FlashCopy 47 record 389
Out of Sync Tracks 397 recovery
overview 37 Global Mirror scenario 274
Global Copy 188, 202 Recovery Point Objective 378
Recovery Point Objective (RPO)
Global Mirror 296
P recreatevg 595
page fault 632 Redbooks Web site 725
parameters 67, 78, 268 Contact us xxx
Consistency Group 261 reestablish Flashcopy relationship
FlashCopy commands 67 B and C volumes 282
paths remote copy events
add paths to existing paths between LLSs 164 SNMP 705
creating 160 remote FlashCopy 52, 65
delete 169 Remote Mirror and Copy 596, 615
pause a Global Mirror session 372 HP-UX 615
pausegmir 290, 320 Remote Mirror and Copy see RMC
pausepprc 399 remove
Global Copy 220 a Global Mirror environment 270
Metro Mirror 144 remove FlashCopy relationship 269
performance replicating data over a distance 244
FlashCopy overview 92 restart application at remote site 282
Global Copy 212 resume
Metro Mirror 130 Global Mirror session 372
persist 389 Metro Mirror 177
Persistent FlashCopy 53 resumegmir 290, 322
physical connection events resumepprc 221
SNMP 702 Metro Mirror 145
planned outage 637 resyncflash 73–74
planned recovery 407 resynchronize target using DS SM 88
point-in-time copy resyncremoteflash 229
creating Global Copy 193 return codes
Index 733
DS CLI 31 trap 102 703
return to local site 284 trap 200 705
reverse restore 54 trap 202 705
reverse source-target relationship using reverseflash 75 trap 210 705
reverseflash 68, 75–76 trap 211 705
revertflash 76–77, 334, 401 trap 212 706
Revertible 402 trap 213 706
revertible 72, 278, 332–333 trap 214 706
RMC 4–5, 515 trap 215 706
commands 28 trap 216 706
Global Copy 5 trap 217 707
Global Mirror 6 traps 701
Metro Mirror 5 Software Information Center 21
VMware ESX server 626 accessing from DS GUI 21
rmflash 77 accessing using Web browser 21
rmgmir 290, 316, 324 start Global Mirror session 259
rmpprc state change logic
Global Copy 217 Global Copy 189
Metro Mirror 142 Storage Complex 10
rmsession 290, 318 adding 498
Rolling Disaster 577 managing 19
rolling disaster 112 procedure to add 19
RPO 378 Storage Facility Image 8
storage servers
add or remove in Global Mirror 268
S Storage Unit 8
scalability Subordinates 400
Global Copy 212 SUN Solaris
Metro Mirror 130 Copy Services 609
scheduled outage 637 FlashCopy with VERITAS Volume Manager 610
script command mode Remote Mirror and Copy with VERITAS Volume Man-
DS CLI 29 ager 612
script mode suspend
DS CLI 29 Metro Mirror 175
SCSI Queue Full 578 Metro Mirror example 175
session 259, 267 Suspended 398
modify 373 switchover 637
pause 372 swpprc 641, 654
view properties 370 symmetrical configuration
view volumes 371 Global Copy 199
setflashrevertible 72–73 Metro Mirror 125
SFI 8 synchronous data replication 244
showckdvol 721 Synchronous PPRC see Metro Mirror
showextpool 721 sysbas 637, 640
showfbvol 721 System i 630
showgmir 290, 346, 391 Copy Services Toolkit 638
showgmiroos 392 Disk Pool 635
Simple Network Management Protocol see SNMP System i 5
simplifying commands logical partitions 631
Metro Mirror 137 System i5
single-level storage 631 backup node 634
single-shot mode cluster 633
DS CLI 29 cluster node 634
SNMP 701 cluster resource group (CRG) 634
Metro Mirror traps 705 device domain 634
notification external storage 630
overview 702 Hardware Management Console (HMC) 631
physical connection events 702 Input output processor (IOP) 632
remote copy events 705 primary node 634
trap 100 702 recovery domain 634
trap 101 702
structure 630–631 Z
subject 632 z/OS Global Mirror 4
zero data loss 378
T
Target Suspended 411
terminology 264
test system 39
tgtinhibit 389
Tier 4 580
Tier 6 580
topology 269
type mmir 388
U
unfreezeflash 78
unfreezepprc 398, 429
unplanned outage 637
user
groups 25
user accounts
DS CLI 25
User ASP 635
user assistance
DS CLI 31
user ID
management 500
V
VDS 604
product components 606
VERITAS Volume Manager 598
view session properties 370
VMware ESX server
Copy Services 616
FlashCopy 617
FlashCopy between 623
FlashCopy between virtual machines 621
FlashCopy within a virtual machine 619
with Remote Mirror and Copy (PPRC) 626
Volume Shadow Copy Services (VSS) 601
volumeaccess 722
volumes
add or remove 267
Metro Mirror 126
VolumeSet level
eRCMF 582
VSS 601
components 601
function 602
W
Windows 598
enlarging extended/spanned volumes 600
extending simple volumes 600
Windows dynamic disks 599
wrkcfgsts 643
WWNN 290
determine for ESS 800 using DS CLI 524
Index 735
IBM System Storage DS8000 Series:
Copy Services in Open
Environments
Copy Services in Open Environments
IBM System Storage DS8000 Series: Copy Services in Open
IBM System Storage DS8000 Series: Copy Services in Open Environments
Environments
Environments
Back cover ®
IBM System Storage DS8000

Series: Copy Services in Open
Environments
Configuration of Copy In today’s highly competitive and real-time environment the

ability to manage all IT needs on a 24 hours a day and 7 days per
INTERNATIONAL
Services in
week basis makes the considerations for the creation of copies TECHNICAL
heterogeneous
and backups a core requirement for any IT deployment. SUPPORT
environments
Furthermore, many clients believe there is a real need to more ORGANIZATION
Metro/Global Mirror pro-actively provide for disaster recovery, and the Copy Services
functions available on IBM TotalStorage DS8000 can form part of
with Incremental
these strategies.
Resync
BUILDING TECHNICAL
Particularly in the open systems environment, Copy Services INFORMATION BASED ON
TPC for Replication functions can provide immediate copies of data, and make it
support and Copy PRACTICAL EXPERIENCE
available to other applications or systems. In effect, this creates
Services with System i a snapshot of the data stored at any given time, now known as a IBM Redbooks are developed
Point-in-Time Copy. Copy Services can also provide local or by the IBM International
remote copies for data retention and disaster recovery purposes. Technical Support
These functions are now known as Metro Mirror, Global Mirror, Organization. Experts from
IBM, Customers and Partners
and Global Copy. This IBM Redbook also covers the 3-site from around the world create
Metro/Global Mirror with Incremental Resync feature and timely technical information
introduces the TotalStorage Productivity Center for Replication based on realistic scenarios.
solution. Specific recommendations
are provided to help you
implement IT solutions more
DS8000 Advanced Copy Services functions can be utilized across effectively in your
the DS8000, DS6000, and some models of the ESS family, and environment.
there are new tools to enable the easier management of these
different storage platforms.
For more information:

ibm.com/redbooks
SG24-6788-02 ISBN 0738496979

Ibm Ds8000

Încărcat de

Informații document

Descriere originală:

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Ibm Ds8000

Încărcat de

Drepturi de autor:

Formate disponibile

Front cover

IBM System Storage DS8000

Metro/Global Mirror with

TPC for Replication support and

Bert Dufrasne Wenzel Kalabzal

IBM System Storage DS8000 Series: Copy Services in

Third Edition (November 2006)

Summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxxi

Chapter 2. Copy Services architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Chapter 3. DS Storage Manager (GUI). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Chapter 4. DS Command-Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

© Copyright IBM Corp. 2005, 2006. All rights reserved. iii

Chapter 5. FlashCopy overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Chapter 6. FlashCopy options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Chapter 7. FlashCopy ordering and activation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Chapter 8. FlashCopy interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

iv IBM System Storage DS8000 Series: Copy Services in Open Environments

Chapter 9. FlashCopy performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Chapter 10. FlashCopy examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Part 4. Metro Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Chapter 11. Metro Mirror overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Chapter 12. Metro Mirror options and configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Chapter 13. Metro Mirror performance and scalability . . . . . . . . . . . . . . . . . . . . . . . . 129

Chapter 14. Metro Mirror interfaces and examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Part 5. Global Copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Chapter 15. Global Copy overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

Chapter 16. Global Copy options and configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 191

vi IBM System Storage DS8000 Series: Copy Services in Open Environments

Chapter 17. Global Copy interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

Chapter 18. Global Copy performance and scalability . . . . . . . . . . . . . . . . . . . . . . . . 211

Chapter 19. Global Copy examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

Part 6. Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Chapter 20. Global Mirror overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243

Chapter 21. Global Mirror options and configuration . . . . . . . . . . . . . . . . . . . . . . . . . 263

Chapter 22. Global Mirror interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Chapter 23. Global Mirror performance and scalability. . . . . . . . . . . . . . . . . . . . . . . . 295

Chapter 24. Global Mirror examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305

Part 7. Metro/Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375

Chapter 25. Metro/Global Mirror overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

Chapter 26. Configuration and setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381

Chapter 27. General Metro/Global Mirror operations. . . . . . . . . . . . . . . . . . . . . . . . . . 395

Chapter 28. Planned recovery scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407

x IBM System Storage DS8000 Series: Copy Services in Open Environments

Chapter 29. Disaster recovery test scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427

Chapter 30. Unplanned scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435

Chapter 31. MGM Incremental Resync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443

Part 8. Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489

Chapter 32. Data migration through double cascading. . . . . . . . . . . . . . . . . . . . . . . . 491

Chapter 33. Interoperability between DS6000 and DS8000 . . . . . . . . . . . . . . . . . . . . . 495

Chapter 34. Interoperability between DS8000 and ESS 800 . . . . . . . . . . . . . . . . . . . . 511

Part 9. Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537

Chapter 35. IBM TotalStorage Productivity Center for Replication . . . . . . . . . . . . . . 539

Chapter 37. IBM TotalStorage Continuous Availability for Windows . . . . . . . . . . . . . 585

Appendix A. Open systems specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591

Appendix B. Copy Services with System i5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629

Appendix C. Simple Network Management Protocol (SNMP) . . . . . . . . . . . . . . . . . . . 701

Appendix D. Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709

Appendix E. CLI migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723

2-1 Logical view of an Storage Complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

© Copyright IBM Corp. 2005, 2006. All rights reserved. xvii

xx IBM System Storage DS8000 Series: Copy Services in Open Environments

© Copyright IBM Corp. 2005, 2006. All rights reserved. xxv