2 Zenterprise Technical Details - ITSOWW2011

yourdotcom
International Technical Support Organization and Authoring Services
zEnterprise Technical Details
www.ibm.com/redbooks
Parwez Hamid Executive IT Consultant IBM UK Limited
2011 IBM Corporation
ibm.com
yourdotcom
Trademarks
The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.
Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.
For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*BladeCenter, DB2, e business(logo), ESCON, eServer, FICON, IBM, IBM (logo), MVS, OS/390, POWER6, POWER6+, POWER7, S/390, System p, System p5, System x, System z, System z9, System z10, WebSphere, z9, z10, z/Architecture, z/OS, z/VM, z/VSE, zSeries
The following are trademarks or registered trademarks of other companies.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.
* All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.
ibm.com
yourdotcom
zEnterprise Technical Details z114

1 2 3 4 5 6
7 8 9 10 11 12 13 IBM System z Family Model, Frame and Processor Drawer Memory Design I/O Infrastruture I/O Subsystem Enhancements Storage Connectivity Features Network Connectivity Features Coupling Link Connectivity and STP I/O Summary Sysplex Consisderations STP Support and Enhancements Crypto RAS Capacity on Demand Capacity and Performance Planning Upgrades Summary
14
15 16 17
3
ibm.com
yourdotcom
IBM System z Family
ibm.com
yourdotcom
IBM System z10 family

IBM System z10 EC (2097) IBM System z10 BC (2098)
Announce 2/08 Server w/ up to 77 PU cores 5 models Up to 64-way Granular Offerings for up to 12 CPs PU (Engine) Characterization
CP, SAP, IFL, ICF, zAAP, zIIP
Announced 10/08 Server w/ 12 cores Single model Up to 5-way CPs High levels of Granularity available
130 Capacity Indicators
PU (Engine) Characterization
On Demand Capabilities
CoD, CIU, CBU, On/Off CoD, CPE
CoD, CIU, CBU, On/Off CoD. CPE
Memory up to 1.5 TB for Server and up to 1 TB per LPAR

16 GB Fixed HSA
Memory up to 256 GB for Server

8 GB Fixed HSA
Channels
Four LCSSs 2 Subchannel Sets MIDAW facility 63.75 subchannels Up to 1024 ESCON channels Up to 336 FICON channels FICON Express2, 4 and 8 zHPF OSA 10 GbE, GbE, 1000BASE-T InfiniBand Coupling Links
Channels
Two LCSSs 2 Subchannel Sets MIDAW facility 63.75 subchannels Up to 480 ESCON channels Up to 128 FICON channels FICON Express2, 4 and 8 zHPF OSA 10 GbE, GbE, 1000BASE-T InfiniBand Coupling Links
Configurable Crypto Express3 Parallel Sysplex clustering HiperSockets up to 16 Up to 60 logical partitions Enhanced Availability Operating Systems
z/OS, z/VM, z/VSE, TPF, z/TPF, Linux on System z
Configurable Crypto Express3 Parallel Sysplex clustering HiperSockets up to 16 Up to 30 logical partitions Enhanced Availability Operating Systems
z/OS, z/OS.e, z/VM, z/VSE, TPF, z/TPF, Linux on System z
ibm.com
yourdotcom
IBM zEnterprise family

IBM zEnterprise 196 (2817) IBM zEnterprise Blade Extension (2458) IBM zEnterprise 114 (2818)
Announced 7/10 Server w/ up to 96 PU cores 5 models Up to 80-way Granular Offerings for up to 15 CPs PU (Engine) Characterization
Announced 7/10 Model 002 for z196 or z114 zBX Racks with:
BladeCenter Chassis N + 1 components Blades Top of Rack Switches 8 Gb FC Switches Power Units Advance Management Modules
Announced 07/11 2 models M05 and M10

Up to 5 CPs
High levels of Granularity available

130 Capacity Indicators
CoD, CIU, CBU, On/Off CoD, CPE
PU (Engine) Characterization
Memory up to 3 TB for Server and up to 1 TB per LPAR

16 GB Fixed HSA
CoD, CIU, CBU, On/Off CoD. CPE
Memory up to 256 GB for Server

8 GB Fixed HSA
Channels
Four LCSSs 3 Subchannel Sets MIDAW facility Up to 240 ESCON channels Up to 288 FICON channels FICON Express8 and 8S zHPF OSA 10 GbE, GbE, 1000BASE-T InfiniBand Coupling Links
Up to 112 Blades
IBM Smart Analytics Optimizer Solution POWER7 Blades IBM System x Blades IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise (M/T 2462-4BX) Operating Systems AIX 5.3 and higher Linux for x Blades Microsoft Windows for x Blades Hypervisors PowerVM Enterprise Edition Integrated Hypervisor for System x
Channels
Two LCSSs 2 Subchannel Sets MIDAW facility Up to 240 ESCON channels Up to 128 FICON channels FICON Express8 and 8S zHPF OSA 10 GbE, GbE, 1000BASE-T InfiniBand Coupling Links
Configurable Crypto Express3 Parallel Sysplex clustering HiperSockets up to 32 Up to 60 logical partitions Enhanced Availability Unified Resource Manager Operating Systems
z/OS, z/VM, z/VSE, z/TPF, Linux on System z
Configurable Crypto Express3 Parallel Sysplex clustering HiperSockets up to 32 Up to 30 logical partitions Unified Resource Manager Operating Systems
z/OS, z/VM, z/VSE, TPF, z/TPF, Linux on System z
ibm.com
4000 3500 3000 2500 MHz 2000 1500 1000 500 0
yourdotcom
z114 continues the CMOS Mainframe heritage

3.5 GHz
3.8 GHz
1.4 GHz 1.0 GHz
625 MHz 139 MHz 413 MHz
1997
Multiprise 2000 Multiprise 2000 1st full-custom Mid-range CMOS S/390 Multiprise 3000 Internal disk, IFL introduced on midrange
7
1999
Multiprise 3000
2002
z800
2004
z890
2006
z9 BC
2008
z10 BC
2011
z114
z800 - Full 64-bit z/Architecture z890 - Superscalar CISC pipeline z9 BC - System level scaling
z10 BC - Architectural extensions Higher frequency CPU z114 Additional Architectural extensions and new cache structure
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) Models, Frame and Processor Drawers
ibm.com
yourdotcom
z114 Overview
Machine Type 2818 2 Models M05 and M10 Single frame, air cooled Non-raised floor option available Overhead Cabling and DC Power Options Processor Units (PUs) 7 PU cores per processor drawer (One for M05 and two for M10) Up to 2 SAPs per system, standard 2 spares designated for Model M10 Dependant on the H/W model - up to 5 or 10 PU cores available for characterization Central Processors (CPs), Integrated Facility for Linux (IFLs), Internal Coupling Facility (ICFs), System z Application Assist Processors (zAAPs), System z Integrated Information Processor (zIIP), optional - additional System Assist Processors (SAPs) 130 capacity settings Memory Up to 256 GB for System including HSA System minimum = 8 GB (Model M05), 16 GB (Model M10) 8 GB HSA separately managed RAIM standard Maximum for customer use 248 GB (Model M10) Increments of 8 or 32 GB I/O Support for non-PCIe Channel Cards Introduction of PCIe channel subsystem Up to 64 PCIe Channel Cards Up to 2 Logical Channel Subsystems (LCSSs) STP - optional (No ETR)
ibm.com
yourdotcom
zEnterprise 114 Models M05 and M10

M/T 2818 Model M05 Air cooled Single Frame Non-raised floor option available 30 LPARs Processor Units (PUs) New processor drawer design (1 processor drawer) 7 per system 2 SAPs standard Up to 5 CPs Up to 5 specialty engines Up to 2 zIIPs/zAAPs 0 spares when fully configured M/T 2818 Model M10 Air cooled Single Frame Non-raised floor option available 30 LPARs Processor Units (PUs) New processor drawer design (2 processor drawers) 14 per system 2 SAPs standard Up to 5 CPs Up to 10 specialty engines Up to 5 zIIPs/zAAPs 2 dedicated spares
When Model M10 (requires the 2nd processor drawer)? > 5 Customer PUs > 120 GB memory > 4 Fanouts for additional I/O connectivity especially PSIFB links Depends - numbers vary for drawers, I/O features and PSIFB links
10 2011 IBM Corporation
ibm.com
yourdotcom
zEnterprise 114 Functions and Features (GA Driver 93 September, 2011)

Two hardware models Up to 10 processors configurable as CPs, zAAPs, zIIPs, IFLs, ICFs, or optional SAPs Up to 130 subcapacity settings across a maximum of 5 CPs Up to 256 GB of Redundant Array of Independent Memory (RAIM) for System Dedicated Spares on the Model M10 Increased capacity processors Out of order instruction execution
Doubled HiperSockets to 32 Additional STP enhancements Doubled Coupling CHPIDs to 128
z114
Improved PSIFB Coupling Link Physical Coupling Links increased to 72 (Model M10) New 32 slot PCIe Based I/O Drawer Increased granularity of I/O adapters New form factor I/O adapters i.e FICON Express8S and OSA-Expres4S Humidity and altimeter smart sensors
Improved processor cache design New and additional instructions On Demand enhancements CFCC Level 17 enhancements Cryptographic enhancements 6 and 8 GBps interconnects 2 New OSA CHPIDs OSX and OSM
Optional High Voltage DC power Optional overhead I/O cable exit zBX-002 with ISAOPT, POWER7, DataPower XI50z and IBM System x Blades NRF Support with either top exit or bottom exit I/O and power Reclassification from general business environment to data center
ibm.com
yourdotcom
zEnterprise zBX Functions and Features
One hardware model zBX is controlled by one specific z196 or z114 Up to 4 Racks (B, C, D and E) 2 BladeCenters Chassis per rack Non-acoustics doors standard Optional Rear Door Heat Exchanger Optional Rear acoustic door Redundant Power, Cooling and Management Modules 10 GbE and 1000BASE-T Network modules 8 Gb FC modules
Advance Management Module 1000BASE-T and 10 GbE TORs
zBX
Up to 112 Blades IBM Smart Analytics Optimizer POWER7 Blades IBM System x Blades IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise HMCs required for Unified Resource Manager Additional zBX owned HMC required if System maintained by Third Party
12
ibm.com
yourdotcom
z114 Model M10 Under the covers

Internal Batteries (optional) Power Supplies 2 x Processor Drawers, Memory & HCAs I/O Drawer
Ethernet cables for internal System LAN connecting Flexible
Service Processor (FSP) cage controller

cards (not shown)
2 x Support Elements
PCIe I/O drawers
FICON & ESCON FQC

13 13
Rear View
Front View
ibm.com
yourdotcom
CEC Drawers Placement
14 14
ibm.com
yourdotcom
Processor / Memory Subsystem Drawers (Model M05 and M10)

One z10 BC Drawer Two z114 Drawers (Model M10)
System resources split between 2 drawers (Model M10) Second processor drawer (Model 10) for: Increased specialty engine capability Increased memory capability Increased I/O capability More coupling links than z10 BC More I/O features than z10 BC Note: Unlike the z196 Books, add/remove/repair of the processor drawer is disruptive
ibm.com
yourdotcom
z114 Processor drawer packaging

Two PU Single Chip Modules (SCMs) & One Storage Control (SC) SCM 10 x Memory Dimms
2 x I/O Fanouts Drw to Drw Interconnect
HCA HCA
inter proc drawer connect
PU SCM
SC SCM
2 x OSC
2 x FSP
HCA
PU SCM
2 x I/O Fanouts
HCA
2x DCA
16
ibm.com
yourdotcom
z114 Processor drawers Infrastructure (M10)

Interconnect (in Brown) - C-shape flex cable SC to SC SMP Signals and Oscillator (OSC) Signals DCA Power
Processor drawer #2
Processor drawer #1
HCA2 or 3s OSC
FSP
HCA2 or 3s
ibm.com
yourdotcom
Oscillator / Fabric Flex Cable
The Oscillator / Fabric Flex Cable which connects Drawer 0 with Drawer 1 Used to transport Oscillator signals (from Drawer 0 to Node1) and to interconnect the Fabric buses
18 18
ibm.com
yourdotcom
z114 Processor drawer #1 (front) with fanout cards
2 Flexible Support Processor (FSP) card slots providing support for the Service Network subsystem 4 fanout card slots providing support for the I/O subsystem and/or coupling 2 x oscillators with Pulse Per Second (PPS)
19
ibm.com
yourdotcom
z114 Processor drawer #2 (front) with fanout cards
2 Flexible Support Processor (FSP) card slots providing support for the Service Network subsystem 4 fanout card slots providing support for the I/O subsystem and/or coupling 2 x oscillators No PPS
20
ibm.com
yourdotcom
Master Oscillator Card for Model 05 with PPS connector
BNC Connector for Pulse Per Second
NTP Ethernet LAN
Note: If the NTP LAN goes through the HMC, PPS can not be used.
21 21
ibm.com
yourdotcom
z114 Model Structure and Upgrades

IFLs Model CPs Unassigned IFLs 0-5 0-10 zAAPs zIIPs ICFs Std. SAPs 2 2 AddI SAPs 0-2 0-2 Spares
M05 M10
0-5 0-5
0-2 0-5
0-2 0-5
0-5 0-10
0 2
Model structure based on number of drawers M05 sparing based on prior Business Class (BC) offerings no dedicated spares M10 sparing based on Enterprise Class (EC) offerings dedicated spares SAP and PU Allocation/Sparing in the M10 Default assignment is one SAP per drawer; one Spare per drawer. Spill and fill CP low to high; spill and fill specialty engines high to low Two defective PUs may cause the default assignment to spill and fill into the second processor drawer. LPAR has the capability to request PU of a specified type to be grouped together in a book/drawer (i.e. LPAR may change the default assignment) Disruptive upgrade from M05 to M10
No model downgrades
Upgrades from z9 BC and z10 BC into either model M05 or M10 Only the M10 will upgrade to z196 Model M15 (Air cooled only)
22 22
ibm.com
yourdotcom
z114 PU Conversions
Specialty engines are converted "any to any", including Additional SAPs and unassigned IFLs. To From.. IFL Unassigned IFL ICF zAAP zIIP Additional SAP IFL x Yes Yes Yes Yes Yes Unassigned IFL Yes x Yes Yes Yes Yes ICF Yes Yes x Yes Yes Yes zAAP Yes Yes Yes x Yes Yes zIIP Yes Yes Yes Yes x Yes Additional SAP Yes Yes Yes Yes Yes x
CP capacity conversions are handled separately from the specialty engines under the following rules:
CP Capacities are converted "any to any"...allow MIPS upgrades, downgrades and horizontal movement. CP Capacities are handled separately from specialty engines, meaning that in the conditions where CP capacity on the target machine is constructed using a lesser amount of CPs, the remaining CPs are not available to be converted to specialty engines. .
23 23
ibm.com
yourdotcom
z114 Conversions (continued)

Any to Any is allowed (upgrades or downgrades) CP Capacities are handled separately from specialty engines. When CP capacity on the target machine is ordered using fewer CPs, the remaining CPs are not available to be converted to specialty engines without a fee.
Examples:
F01 E01 D01 C01 B01 A01 1-way F02 E02 D02 C02 B02 A02 2-way F03 E03 D03 C03 B03 A03 3-way F04 E04 D04 C04 B04 A04 4-way F05 E05 D05 C05 B05 A05 5-way
2-way to 1-way capacity downgrade The PU in the green column must maintain its high water mark and can not be purchased as a specialty engine. 2-way to 1-way capacity upgrade The high water mark is now in the red column and the PU in the green column can be used during the purchase of a specialty engine.
24 24
ibm.com
yourdotcom
IBM System z Business Class Configuration Comparisons (1 of 3)

z9 BC R07
Uniprocessor Performance z/OS Capacity Total System Memory Configurable Engines Dedicated Spares Configurable CPs LPARS/LCSS HiperSockets I/O Cages/Drawers I/O slots per Cage/Drawers FICON Channels OSA Ports (10GbE/1GbE) ESCON Channels STI (z9), IFB (z10) Bandwidth PCIe (z114 BC) Bandwidth ICB-4/ISC-3/PSIFB zIIP/zAAP Maximum Qty IFL Maximum Qty Capacity Settings Upgradeable 3 6 20 1 16 64 16/32 240 2.7 GB/sec 16/48/0 3 7 53 7 0 1-3 15/1 16 1 28 112 24/48 420 470 MIPS 26-172 MIPS 193-1748 MIPS 64 GB 7 0 0-4 30/2
z9 BC S07
z10 BC E10
673 MIPS 26-2760 MIPS 256 GB 10 0 0-5 30/2 16 Up to 4 8 128 48/96 480 6.0 GB/sec 12/48/12 5 10 130 Upgrade to z114
z114 M05
782 MIPS
z114 M10
26 - 3139 MIPS 128 GB 5 0 0-5 30/2 32 Up to 3(1) 8/32(2) 128(3) 48/96 240(4) 6.0 GB/sec 8.0 GB/sec (5) 0 /48/8 -16(6) 0(5)/48/16 - 32(7) 2 5 130 5 10 130 Up to 3(1) 256 GB 10 2
Upgrade to z10 and z114
Upgrade From M10 to z196 (M15, Air cooled only)
25
See next chart for foot notes
ibm.com
yourdotcom
Notes for Configuration comparisons chart (2 of 3)

(1)
Up to 3 channel drawers standard, a combination of I/O drawers and PCIe I/O drawers as defined. Up to 4 I/O drawers via RPQ
M05 I/O Drawer 0 0 0 1 1 PCIe I/O Drawer 0 1 2 0 1 I/O Drawer 0 0 0 1 1 1 2 2 3 4

(2) (3) (4) (5) (6) (7) 26
M10 PCIe I/O Drawer 0 1 2 0 1 2 0 1 0
0 1 0 0
2 2 3
3rd and 4th I/O drawers are offered via an RPQ
8 card slots per I/O drawer, 32 per PCIe I/O drawer FICON count is based on 2 PCIe I/O drawers or 4 I/O drawers Limit of 240 ESCON channels is consistent with Statement of Direction Limit of 0 ICB-4 links is consistent with Statements of Direction 8 ports of 12x, 16 ports of 1x PSIFB links available on model M05 based on 4 HCA capabilities 16 ports of 12x, 32 ports 1x PSIFB links available on model M10 based on 8 HCA capabilities
ibm.com
yourdotcom
IBM System z Config Comparisons, z114 vs. z196 Model M15 (3 of 3)

z114 M05
Uniprocessor Performance z/OS Capacity Total System Memory Configurable Engines Configurable CPs LPARS/CSS HiperSockets I/O Cages/Drawers I/O slots per Cage/Drawers FICON Channels OSA Ports (10GbE/1GbE) ESCON Channels STI (z9), IFB (z10) Bandwidth PCIe (z114/z196) Bandwidth ICB-4/ISC-3/PSIFB zIIP/zAAP Maximum Qty IFL Maximum Qty Capacity Settings Upgradeable Up to 3(1) 8/32(2) 128(3) 48/96 240(4) 6.0 GB/sec 8.0 GB/sec 0(5)/48/8 -16(6) 2 5 130 0(5)/48/16 - 32(7) 5 10 130 782 MIPS 26 - 3139 MIPS 120 GB 5 0-5 30/2 32 Up to 3(1) 248 GB 10
z1114 M10
z196 Model M15

1200 MIPS 240 13,660 MIPS 704 GB 15 0 - 15 60/4 32 3/6 28/8/32(2) 256 48/96 240 6.0 GB/sec 8.0 GB/sec 0(5)/48/16 - 32 7 15 60 z196 M32, M49, M66, M80 MRU and Water Cooled
Upgrade from M10 to z196 (M15, MRU cooled only)
See previous chart for foot notes

ibm.com
yourdotcom
IBM zEnterprise 114 (z114) Processor Unit Design
28
ibm.com
yourdotcom
z114 PU/SC Single Chip Module (SCM) Components

Heatsink
Thermal Interface Material

PU Chip
LGA
Processor drawer with 2 PU SCMs, 1 SC SCM Memory DIMMS and DCA Power
ibm.com
yourdotcom
z114 SCM Vs z196 MCM Comparison Same PU and SC Chip z114 SCMs
PU SCM 50mm x 50mm in size fully assembled
Quad core chip with 3 and 4 active cores 2 PU SCMs for M05 and 4 PU SCMS for M10 PU Chip size 23.498 mm x 21.797 mm
z196 Multi Chip Module (MCM)

MCM 96mm x 96mm in size 6 PU chips per MCM Quad core chips with 3 or 4 active cores
PU Chip size 23.498 mm x 21.797 mm
SC SCM 61mm x 61mm in size fully assembled

1 SC SCM for M05, 2 SC SCMs for M10 96 MB L4 cache per chip SC Chip size 24.427 mm x 19.604 mm
2 SC chips per MCM

96 MB L4 cache per chip SC Chip size 24.427 mm x 19.604 mm Up to 4 MCMs for System
Single PU Chip without heatsink
Single SC Chip without heatsink
PU 2
PU 1
PU 0
S10 SC 1 S11 PU 3 PU 4 SC 0
S00 S01 PU 5
30
ibm.com
yourdotcom
z114 PU Chip Details

12S0 45nm SOI Technology 13 layers of metal Chip Area 512.3mm^2 23.5mm x 21.8mm 8093 Power C4s 1134 signal C4s 1.4 Billion Transistors 4 Core Design 3.8 GHz Full out-of-order capability 1.5MB L2 cache 12MB shared L3 cache Two Co-Processors Data Compression & encryption Memory controller Supports 5-channel DDR3 RAIM GX Controller 3 PLLs, 5+ frequency domains 5 unique chip voltage supplies
31
MC IOs
Core 0
L3 Cache (12MB) L3B
Core 2
GX IOs
L2 (1.5MB) MCU CoP L2 (1.5MB) MC IOs L3_0 Controller L3_1 Controller L3B Core 1 L3 Cache (12MB)
L2 (1.5MB) CoP GX L2 (1.5MB)

GX IOs
Core 3
ibm.com
yourdotcom
z114 SC Chip Details
12S0 45nm SOI Technology 13 layers of metal Chip Area 478.8mm^2 24.4mm x 19.6mm 7100 Power C4s 1819 signal C4s
1.5 Billion Transistors 1 Billion cells for eDRAM eDRAM Shared L4 Cache 96 MB per SC chip 6 CP chip interfaces
3 Fabric interfaces
1 used
2 clock domains 5 unique chip voltage supplies
2 used
32 32
ibm.com
yourdotcom
z114 System Topology

2-drawer System Up 14 Processing Units at 3.8 GHz 3 or 4 cores active, same as z196 Up to 16 HCA ports (M10) Support for Legacy and PCIe HCAs Up to 4 Memory Controllers
FP2
5 DIMMs
5 DIMMs
GX06 GX02
GX1 GX0
Drawer #1
GX07 GX01
GX1 GX0
CP02
SC0
CP2
SC00 CP1
SC0
CP01
Up to 20 DIMM slots 5-channel Memory RAIM protection 4-level cache hierarchy Split 64K I-L1 and 128K D-L1 per core 1.5 MB L2 per core 12 MB on chip L3 half that of z196 96 MB node level L4 half that of z196 eDRAM (embedded) caches L3, L4 and SP Key Cache
Color legend: Same as in z196 models Different 33 2011 IBM Corporation
PSI Redundant PSI Redundant

FP2
CP12
SC0
CP2
SC10 CP1
SC0
CP11
GX1 GX0
GX1 GX0
GX16 GX12
5 DIMMs
Drawer #2
GX17 GX11
5 DIMMs
ibm.com
yourdotcom
z114 Internal System Structure Compared to the z196

BOOK
Mem2 Mem1 Mem0
Topology
GX2 GX6 GX7
GX1
GX0
PU2
PU1
CP0
Book
FBC
FBC
Book
FBC0 FBC1 FBC2
SC1
SC0
FBC0 FBC1 FBC2
FBC FBC
FBC
Book CP3 CP4 CP5
FBC
Book
GX3
GX4
z114 is built from subset of the z196 design and chipset Half the L3 and L4 cache sizes Half the chip to chip bandwidth
PSI
GX5
34
ibm.com
yourdotcom
z114 Cache Hierarchy
35 35
ibm.com
yourdotcom
Cache Hierarchy z196 (4 books) vs. z114 (2 processor drawers) Comparison

4 L4 Caches 2 L4 Caches
192MB Shared eDRAM L4

24MB Shr eDRAM L3
L2 L1 L2 L1 L2 L1 L2 L1
96MB Shared eDRAM L4

12MB Shr eDRAM L3
L2 L1 L2 L1 L2 L1 L2 L1
6 L3s, 24 L1 / L2s
L2 L1
24MB Shr eDRAM L3

L2 L1 L2 L1 L2 L1
2 L3s, 7 L1 / L2s
12MB Shr eDRAM L3

L2 L1 L2 L1 L2 L1 L2 L1
L1:
64KI + 128KD 8w DL1, 4w IL1 256B line size Private 1.5MB Inclusive of L1s 12w Set Associative 256B cache line size Shared 24MB Inclusive of L2s 12w Set Associative 256B cache line size 192MB Inclusive 24w Set Associative 256B cache line size
L1:
64KI + 128KD 8w DL1, 4w IL1 256B line size Private 1.5MB Inclusive of L1s 12w Set Associative 256B cache line size Shared 12MB Inclusive of L2s 12w Set Associative 256B cache line size 96MB Inclusive 24w Set Associative 256B cache line size
L2
L2
L3
L3
L4
L4
z196
36 36 2011 IBM Corporation
z114
ibm.com
yourdotcom
z114 Cache Structure

On the processor chip there are 3 levels of cache
L1 I Instruction Cache of 64KB L1 D Data Cache of 128KB L2 Private Cache of 1.5MB (per core) L3 Share Cache of 12MB
L3 Cache
12MB eDRAM Gear Ratio: 2:1 dataflow, 4:1 control flow 2 address slices, address bit 55 selection 16 octword cache interleaves Address and Data bit stacking Store-in cache, store dual-piped per L3 slice 16B 2:1 store/fetch buffers planned between L2&L3 Extension of z10 cache coherency scheme CP/SC merged transaction bussing (addr/control + data) IO fetch reserved cache slots
37 37
ibm.com
yourdotcom
z114 Cache Structure L4 Cache

L4 directory is used to filter snoop traffic from the remote drawer Enhanced Fabric Protocol for improved latency and cache management Horizontal Cache Persistence for local cache extension LRU install on available cache slots in the remote L4 cache 96 MB eDRAM per SC chip, 24 Set Associatively. Gear ratio: 2:1 dataflow, 4:1 control flow. Synchronous fabric port 2 address sliced L4 Directories within the SC chip, address bit 54 selection 8 256B cache banks per logical directory z10 derived cache and storage coherency scheme
38 38
ibm.com
yourdotcom
zEnterprise System PU core

Each core is a superscalar, out of order processor with these characteristics: Six execution units 2 fixed point (integer), 2 load/store, 1 binary floating point, 1 decimal floating point Up to three instructions decoded per cycle (vs. 2 in z10) 211 complex instructions implemented by multiple internal operations 246 of the most complex z/Architecture instructions are implemented via millicode Up to five instructions/operations executed per cycle (vs. 2 in z10) Execution can occur out of (program) order Memory address generation and memory accesses can occur out of (program) order Special circuitry to make execution and memory accesses appear in order to software Each core has 3 private caches 64KB 1st level cache for instructions, 128KB 1st level cache of data 1.5MB L2 cache containing both instructions and data
39
ibm.com
yourdotcom
zEnterprise System Out of Order (OOO) Value

OOO yields significant performance benefit for compute intensive apps through Re-ordering instruction execution Later (younger) instructions can execute ahead of an older stalled instruction Re-ordering storage accesses and parallel storage accesses OOO maintains good performance growth for traditional apps
Instrs 1 2 3 4 5
In-order core execution
Out-of-order core execution
L1 miss
L1 miss
Time
Execution Storage access
Time
40
ibm.com
yourdotcom
zEnterprise System Out of Order Detail

Out of order yields significant performance benefit through Re-ordering instruction execution
Instructions stall in a pipeline because they are waiting for results from a previous instruction or the execution resource they require is busy In an in-order core, this stalled instruction stalls all later instructions in the code stream In an out-of-order core, later instructions are allowed to execute ahead of the stalled instruction
Re-ordering storage accesses

Instructions which access storage can stall because they are waiting on results needed to compute storage address In an in-order core, later instructions are stalled In an out-of-order core, later storage-accessing instructions which can compute their storage address are allowed to execute
Hiding storage access latency

Many instructions access data from storage Storage accesses can miss the L1 and require 10 to 500 additional cycles to retrieve the storage data In an in-order core, later instructions in the code stream are stalled In an out-of-order core, later instructions which are not dependent on this storage data are allowed to execute
41
ibm.com
yourdotcom
zEnterprise System Compression and Cryptography Accelerator
Data compression engine

Static dictionary compression and expansion Dictionary size up to 64KB (8K entries) Local 16KB cache per core for dictionary data
Core 0 2nd Level Cache IB OB TLB
Core 1
CP Assist for Cryptographic Function (CPACF)

Enhancements for new NIST standard Complemented prior ECB and CBC symmetric cipher modes with XTS, OFB, CTR, CFB, CMAC and CCM New primitives (128b Galois Field multiply) for GCM
TLB
OB
IB
Cmpr Exp
16K
16K
Cmpr Exp
Accelerator unit shared by 2 cores

Independent compression engines Shared cryptography engines
Crypto Cipher
Crypto Hash
42
ibm.com
yourdotcom
z196 and z114 Comparison z196

Nodal Topology Number of books / drawers Max # of PUs Number of chips per Book/Drawer 1st level cache (PU core) 2nd level cache (PU core) 3rd level cache (PU nest) 4th level cache (SC) # of PU L3 pipes # of SC cache pipes Memory DIMMs Max System Memory Size Fully Interconnected 4 books 96 6 PUs, 2 SCs 64KB I, 128KB D 1.5MB private per core (L2) 24MB Shared per 4 cores (L3) 192MB Shared per node (SC L4) 2 Main, 2 Store 1 Main (per SC) x81 DIMMs 3 Terabytes on 120 DIMMs
z114
Bi-Nodal 2 drawers 14 2 PUs, 1 SC 64KB I, 128KB D 1.5MB private per core (L2) 12MB Shared per 4 cores (L3) 96MB Shared per node (SC L4) 1 Main, 1 Store 1 Main x81 DIMMs 256 Gigabytes on 20 DIMMs
43
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) Memory Design
44
ibm.com
yourdotcom
z114 RAIM Memory

Level 4 Cache
16B 16B 16B 16B
Memory technology introduced on the z196 is used on the z114 Redundant Array of Memory (RAIM) which in the Disk industry is known as RAID Protection from Unscheduled Incident Repair Actions (UIRAs) caused by a DIMM failure
DIMM failures include all components on the DIMM Portions of the memory controller or card failure isolated to one memory channel
Key Cache
Key Cache
MCU 0
2B
MCU 1
2B
DATA CHECK
DATA CHECK ECC RAIM Parity
Flexible memory option not available on z114 Used for Enhanced Book Availability on z196
Extra column provides RAIM function
45
ibm.com
yourdotcom
z114 Memory Topology
10 DIMMSs/drawer
No cascading
All DIMMs in drawer must be the same
DIMMs plugged in quints 5 DIMMs
46 46
ibm.com
yourdotcom
z114 Memory Details

Model M05 M10 M10 Physical base memory
M05 = 40GB, 24GB for concurrent ordering before a physical (disruptive) card change (minus HSA/RAIM) M10 = 80GB, 56GB for concurrent upgrade ordering before a physical (disruptive) card change (minus HSA/RAIM)
Increment (GB) 8 8 32
Standard (GB) 8 120 16 120 152 - 248
Plan Ahead (GB) 24 120 56 120 152 - 248
FC1903 - Memory Capacity Increment Used to determine memory billing for new memory being added server configurations Indicates 8GB (or 32 GB in larger configurations) LICCd increments of Memory Capacity. Charged by increments when Plan Ahead memory is enabled (conversion) Subsequent memory upgrade orders will use up the Plan Ahead memory first
FC 1904 - 16GB Feature Converted Memory Used to determine billing for memory on MES upgrades coming from System z9 and z10 servers.. FC1993 Preplanned Memory Tracks the quantity of 8GB physical increments. Charged (half price) when physical memory is installed
ibm.com
yourdotcom
z114 Model M05 Memory Features

10 x 4 GB DIMMs 10 x 16 GB DIMMs
Feature Size
10 x 8 GB DIMMs
Plan Ahead Memory Pre-plugged memory based on Feature Size Feature Size target capacity specified by the customer. 8 32 Enabled by LICCC, concurrently. FC1993 tracks the quantity of 8GB 16 40 physical increments. 24 48 Charged (half price) when physical memory is installed 56 FC1903 generally indicates 8GB (or 32 GB in larger configurations) LICCd increments of Memory Capacity. Charged by increments when Plan Ahead memory is enabled Subsequent memory upgrade orders will use up the Plan Ahead memory first
64 72 80 88 96 104 112 120
Physical memory upgrades are DISRUPTIVE

ibm.com
yourdotcom
z114 Model M10 Memory Features
32 GB increments
4 GB/4GB
Feature Size
4GB/8GB 8GB/4GB
Feature Size
8GB/8GB
Feature Size
4GB/16GB 16GB/4GB
Feature Size
8GB/16GB 16GB/8GB
Feature Size
16GB/16GB
Feature Size
16 24 32 40 48 56
64 72 80 88
96 104 112 120
152
184
216 248
Physical memory upgrades are DISRUPTIVE

ibm.com
yourdotcom
z114 Memory Offerings

FC GB Increment Dial Max 3609 3610 3611 3612 3613 3614 3615 3616 3617 3618 3619 3620 3621 3622 3623 3624 3625 3626 3627 50 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 152 184 216 248 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 32 32 32 32 2011 IBM Corporation 120 56 24 M05 Dimm (GB) 4 4 4 8 8 8 8 16 16 16 16 16 16 16 16
Memory upgrades within the same color (except white) are concurrent without the need for Memory Plan Ahead.
M10 # plugged 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 152 120 88 Dial Max N/A 56
(2 CPC drawers)
Dimm (GB) N/A 4/4 4/4 4/4 4/4 4/4 4/4 4/8 4/8 4/8 4/8 8/8 8/8 8/8 8/8 4/16 8/16 16/16 16/16
# plugged N/A 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10 10/10
N/A
184 248
ibm.com
yourdotcom
System z HSA Comparisons

HSA significantly larger than pre-z10 Servers Fixed HSA, does not affect customer purchased memory Size of HSA on prior Servers (dependant on defined configuration)
Multiprise 2000 9672 G4 Multiprise 3000 9672 G5/G6 z800 z900 z890 z990 z9 BC z9 EC z10 BC z10 EC z196 z114 From 12 MB up to 40 MB From 48 up to 64 MB From 38 MB up to 136 MB From 64 MB up to 192 MB From 160 MB up to 256 MB From 288 MB up to 512 MB From 768 MB up to 1.9 GB From 1 GB MB up to 2 GB From 896 MB up to 2.7 GB From 1.2 GB up to 4.2 GB 8 GB Fixed 16 GB Fixed 16 GB Fixed 8 GB - Fixed
HSA Estimator on Resource Link not relevant
51
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) I/O Infrastructure
52
ibm.com
yourdotcom
Glossary for System z I/O on z196 and z114

Acronym
N/A N/A N/A N/A PCI-IN
Full Name
I/O drawer I/O cage PCIe switch PCIe I/O drawer PCIe interconnect
Description / Comments
I/O drawer introduced with z10 BC and also supported on z196 and z114; has 8 I/O card slots I/O cage available since z900 (not supported on z10 BC or z114); has 28 I/O card slots Industry standard PCIe switch Application-specific integrated circuit (ASIC) used to fanout (or multiplex) the PCI bus to the I/O cards within the PCIe I/O drawer New I/O drawer that supports PCIe bus I/O infrastructure; has 32 I/O card slots Card in the PCIe I/O drawer that contains the PCIe switch ASIC z10 uses IFB-MP; z9 uses STI-MP Card on front of processor book that supports PCIe Gen2 bus; used exclusively to connect to the PCIe I/O drawer; PCIe fanout supports FICON Express8S and OSA-Express4S Used instead of an HCA2-C fanout for I/O which continues to support the cards in the I/O cage and I/O drawer For 1x InfiniBand at unrepeated distances up to 10 km; 5 Gbps link data rate; 4 ports per fanout; may operate at 2.5 Gbps or 5 Gbps. Based upon capability of DWDM. Exclusive to z196 and z114; can communicate with an HCA2-O LR fanout; third generation Host Channel Adapter For 12x InfiniBand at 150 meters; supports 12x IFB and 12x IFB3 protocols; increased service times when using 12x IFB3 protocol; 6 GBps link data rate; two ports per fanout; can communicate with an HCA2-O fanout on z196 or z10; cannot communicate with an HCA1-O fanout on z9; third generation Host Channel Adapter
N/A
PCIe fanout
HCA3 or HCA3-O LR HCA3 or HCA3-O
HCA3-O LR fanout for 1x IFB
HCA3-O fanout for 12x IFB
53
ibm.com
yourdotcom
Glossary for System z InfiniBand Coupling

Acronym AID CIB HCA PSIFB 12x IFB 1x IFB 12x IFB3 Type HCA1-O fanout HCA2-C fanout HCA2-O fanout HCA2-O LR fanout PCIe fanout HCA3-O fanout HCA3-O LR fanout MBA fanout
Full name Adapter identification Coupling using InfiniBand Host Channel Adapter Parallel Sysplex using InfiniBand 12x InfinBand 1x InfiniBand 12x InfiniBand3 z10 N/A (z9 only) Copper Connects to I/O Cage (z10 EC only) or I/O Drawer Optical - Coupling 12x InfiniBand Optical - Coupling 1x InfiniBand N/A N/A N/A Copper - Coupling (ICB-4)
Comments HCA fanout has AID instead of a PCHID CHPID type z196, z114, z10, System z9 Path for communication InfiniBand Coupling Links 12 lanes of fiber in each direction Long Reach - one pair of fiber Improved service times of 12x IFB on HCA3-O z196/z114 N/A (z9 only) Copper - Connects to I/O Cage (z196 only) or I/O Drawer Optical - Coupling 12x InfiniBand Optical - Coupling 1x InfiniBand Copper - Connects to PCIe Drawer Optical - Coupling 12x InfiniBand Optical - Coupling 1x InfiniBand N/A
ibm.com
yourdotcom
System z I/O Interface Evolution
z9
Extended Link (processor to IO Cage) eSTI (IBM)
z10 z196
InfiniBand DDR (Industry Standard)
z196 z114 Sep 2011

PCI Express (Ind. Std.)
IO Cage Backplane
mSTI (IBM)
mSTI (IBM)
PCI Express (Ind. Std.)
Coupling Link
ISC (IBM)
InfiniBand DDR (Industry Standard)
InfiniBand DDR (Ind. Std.)
55
ibm.com
yourdotcom
System z I/O Subsystem Internal Bus Interconnect Speeds (GBps)

PCIe z196/z114 Sep 2011 InfiniBand z10/z196/z114
8 GBps 6 GBps 2.7 GBps 2 GBps 1 GBps

STI: Self-Timed Interconnect
STI z9
STI z990/z890
STI z900/z800
ibm.com
yourdotcom
PCIe Internal I/O Infrastructure

System z continues the I/O evolution with a new internal I/O infrastructure change on the platform to continue to support the industry direction for new, high performance I/O Increase the internal bandwidth of I/O infrastructure backbone from InfiniBand (6 GBps) to PCIe (8 GBps) Allow for future I/O performance growth requirements Provides future power management capability to reduce overall power consumption Adopt industry standard technology (PCIe) System z strategic direction to utilize industry standards without compromising robustness and System z qualities of service. Enables improved I/O capability and flexibility by leveraging the technology investments of the industry Allows for the future possibility of integration of industry standard, native PCIe adapters and accelerators
57
ibm.com
yourdotcom
PCIe I/O drawer 1 Of 2

Compact Two PCIe I/O drawers occupy the same space as one I/O cage Increased granularity Two FICON Express8S channels per feature One or two OSA-Express4S ports per feature I/O card density 14% more capacity 32 I/O card slots Peripheral Component Interconnect Express (PCIe Gen2) 8 GBps bus Infrastructure from processor book/drawer to I/O cards Reduced power consumption Designed for improved RAS Symmetrical, redundant cooling across all cards and power supplies Temperature monitoring of critical Application-specific integrated circuits (ASICs) Concurrent add/repair of PCIe I/O drawer and adapters
58
ibm.com
yourdotcom
PCIe I/O drawer 2 of 2

Designed to Support concurrent add and repair Support for a new industry standard I/O Attachment of select industry standard PCIe cards Provide a balanced system Designed to provide improved performance for new and traditional workloads Lower power requirements while increasing connectivity for all workloads Increased infrastructure bandwidth I/O bus infrastructure data rate up to 8GBps Positioned for higher bandwidth in the future Power Save Mode Power can be reduced on unused ports and unused area of the drawer Fewer ports per I/O card, but support for four times as many slots. Intended for new I/O features (FICON Express8S, OSA-Express4S) Legacy I/O technology (ESCON, ISC-3, PSC, Crypto Express3 and carry forward) still supported on I/O Drawer (FC4000)
59
ibm.com
yourdotcom
z196/z114 Internal I/O Infrastructure Overview

The z196/z114supports two different internal I/O infrastructures The z196 supports: InfiniBand based infrastructure for I/O cages and I/O drawers PCIe based infrastructure for PCIe I/O drawers with a new form factor drawer and I/O features The z114 supports: InfiniBand based infrastructure for I/O drawers PCIe based infrastructure for PCIe I/O drawers with a new form factor drawer and I/O features
60
ibm.com
yourdotcom
z114 and z196 at GA2 support two different internal I/O infrastructures
The InfiniBand I/O infrastructure first made available on z10 InfiniBand fanouts supporting the current 6 GBps InfiniBand I/O interconnect InfiniBand I/O card domain multiplexers with Redundant I/O interconnect in: The 14U, 28-slot, 7-domain I/O cage (z196 only) The 5U, 8-slot, 2-domain IO drawer (z114 and z196) Selected legacy I/O feature cards Carry forward and new build And a new PCI Express 2 I/O infrastructure PCIe fanouts supporting a new 8 GBps PCIe I/O interconnect PCIe switches with Redundant I/O interconnect in for I/O domains in a new 7U, 32-slot, 4-domain I/O drawer (z114 and z196 GA2) New FICON Express8S and OSA-Express4S I/O feature cards Based on selected industry standard PCIe I/O Designed to:
Reduce purchase granularity (fewer ports per card) Improve performance Increase I/O port density Save energy
61
ibm.com
yourdotcom
8-slot I/O Drawer (Introduced with z10 BC)

Front
DCA
I/O - A I/O - B I/O - B I/O - A 5U
Supports all z10 BC and z196 GA1 I/O and Crypto Express3 cards Supports 8 I/O cards, 4 front and 4 back, horizontal orientation, in two 4-card domains (shown as A and B) Requires two IFB-MP daughter cards, each connected to a 6 MBps InfiniBand interconnect to activate both domains.
Rear
DCA
I/O - B I/O - A IFB-MP A 5U IFB MP B
RII
I/O - B I/O - A
To support Redundant I/O Interconnect (RII) between the two domains, the two interconnects must be from two different InfiniBand fanouts. (Two fanouts can support two of these drawers.) Concurrent add, repair. Requires 5 EIA Units of space (8.75 inches 222 mm)
62
ibm.com
yourdotcom
New 32 slot PCIe I/O drawer for z114 and z196

Front
Supports only the new PCIe I/O cards introduced with z114 and z196 GA2. Supports 32 PCIe I/O cards, 16 front and 16 rear, vertical orientation, in four 8card domains (shown as 0 to 3). Requires four PCIe switch cards ( ), each connected to an 8 MBps PCIe I/O interconnect to activate all four domains. To support Redundant I/O Interconnect (RII) between front to back domain pairs 0-1 and 2-3 the two interconnects to each pair must be from 2 different PCIe fanouts. (All four domains in one of these cages can be activated with two fanouts.) Concurrent field install and repair. Requires 7 EIA Units of space (12.25 inches 311 mm)
7U Domain 0
Domain 2
Rear
7U Domain 3 Domain 1
ibm.com
yourdotcom
zEnterprise I/O Drawer comparison

I/O Infrastructure
PROCESSOR DRAWER PROCESSOR DRAWER
PROCESSOR
PROCESSOR
I/O Drawer FC4000

(same as z10 BC)
PCIe I/O Drawer FC4003
GX I/O Bus
GX I/O Bus
HCA
HCA
EXTENDED LINKS
in 30
midplane
30 in
midplane
FANOUT & FAILOVER
FAILOVER
FANOUT & FAILOVER
8.75 in
CAGE BUSES
I/O Cards
DRAWER
I/O Cards
22.75 in
12.25.75 in
22.75 in
8 horizontal slots 4 Front/4 Back 32 FICON Ports 2 DOMAINS
32 vertical slots 16 Front/16 Back 64 FICON Ports 4 DOMAINS
64
ibm.com
yourdotcom
Top views I/O cage, I/O drawer, PCIe I/O drawer Domains
I/O cage 28 slots 7 I/O domains
1A 2B 3A 4B
I/O drawer 8 slots 2 I/O domains

DCA 2 DCA 1 I/O-A
PCIe I/O drawer 32 slots 4 I/O domains

0 0 0 0
PCIe interconnect
RII
B W I R N G D E
25E 26F 27E 28F 17G 18G 19G 20G
RII
I/O-B I/O-A
1 1 1 1
PCIe interconnect
IFB-MP
5A 6B 7A 8B 9C 10D 11C 12D
I/O-B I/O-B G I/O-A
IFB-MP B IFB-MP A
I/O-A I/O-B
21E 22F 23E 24F
Front - 4
F
Rear - 4
C
13C 14D 15C 16D
2 2 2 2
PCIe interconnect
RII
0 0 0 0 FSP-1, 1
1 1 1 1 FSP-1, 2 3 3 3 3
PCIe interconnect
Redundant I/O interconnect = RII
2 2 2 2
3 3 3 3
Front 16
65
Rear 12
Front 16
Rear 16
ibm.com
yourdotcom
I/O Drawer
No changes to as used on 2098,and 2817 7u I/O drawer The addition or removal of a drawer is a concurrent MES IFB-MP cards as used on I/O Cage DCA's are unchanged from 2098 z10-BC Each houses 8 I/O cards over 2 domains Min/Max is 0/4 for new build and 0/4 for MES All card assemblies in the can be added concurrently.
66 66
ibm.com
yourdotcom
PCIe I/O Drawer

7u I/O drawer The PCIe drawer minimum/maximum is 0/2 The addition or removal of a PCIe drawer is a concurrent MES Each PCIe houses 32 I/O cards over 4 domains The PCIe drawer is for new PCIe based I/O only. ESCON, ISC-3 and PSC24V cannot be plugged in a PCIe drawer All card assemblies in the PCIe can be added concurrently
67 67
ibm.com
yourdotcom
z114 Fanout Usage for I/O (Mixed PCIe and I/O drawers)
PCIe Fanout Support
1 PCIe drawer = 2 PCIe fanouts 2 PCIe drawers = 4 PCIe fanouts Processor drawer 0
Memory
PU PU PU
HCA2-C Fanout Support

1 I/O drawer = 2 HCA2-C 2 I/O drawer = 2 HCA2-C 3 I/O drawer = 4 HCA2-C (RPQ) 4 I/O drawer = 4 HCA2-C (RPQ)
SC1, SC0 (FBC)

PU PU PU
PCIe (8x)
PCIe (8x)
HCA2-C (6x)) HCA2-C (6x))
IO Drawer 2 PCIe switch PCIe switch PCIe switch PCIe switch

IFB-MP IFB-MP
I/O Drawer 1
4 GB/s PCIe Gen2 x8
Channels
FICON Express8S OSA-Express4S 10 GbE
Ports OSA-Express3 1000BASE-T
ESCON
68
FPGA
ESCON
FPGA
Max of 4 fanouts per processor drawer !

ESCON ESCON ESCON
2GBps mSTI
ibm.com
yourdotcom
z114 M05 Redundant I/O Interconnect Example One PCIe Drawer

P C I e F A N O U T F A N O U T P C I e
Different PCIe Fanouts Support Domain Pairs: 0 and 1 2 and 3 Normal operation each PCIe interconnect in a pair supports the eight I/O cards in its domain. Backup operation: One PCIe interconnect supports all 16 I/O cards in the domain pair.
PCIe switch cards ( )
Front
Rear
Domain 3 Domain 0 Domain 2
Domain 1
69
ibm.com
yourdotcom
z114 Fanout Usage for the I/O and PCIe drawers

Number and types of drawers Zero I/O drawers One I/O drawer One PCIe I/O drawer Two I/O drawers One I/O drawer + One PCIe drawer Three I/O drawers Two PCIe drawers Two I/O drawers + One PCIe drawer Four I/O drawers (RPQ) One I/O drawer + Two PCIe drawers Model 05 0 2 x HCA2-C 2 x PCIe fanouts 2 x HCA2-C 2 HCA2-C & 2 x PCIe fanouts 4 x HCA2-C 4 x PCIe fanouts 2 HCA2-C & 2 x PCIe fanouts 2 x HCA2-C N/A not enough fanouts Model 10 0 2 x HCA2-C 2 x PCIe fanouts 2 x HCA2-C 2 HCA2-C & 2 x PCIe fanouts 4 x HCA2-C 4 x PCIe fanouts 2 HCA2-C & 2 x PCIe fanouts N/A no drawer slots available 2 HCA2-C & 4 x PCIe
Notes: Each HCA2-C and PCIe fanout has 2 ports Each I/O drawer has 2 I/O domains (4 I/O features per domain). All 4 domains are plugged to the HCA2-C Each PCIe drawer has 4 I/O domains (8 I/O features per domain). 2 domains at a time are plugged to the PCIe fanout 4th I/O drawer uses one of the processor drawer locations
ibm.com
yourdotcom
z114 HCA Fanout Cards Summary

Description HCA2-C copper fanout HCA2-O 12x IB-DDR HCA1-O 12x IB-SDR HCA2-O LR 1x IB-DDR Carry Forward only PCIe copper fanout HCA3-O LR 1x IFB HCA3-O 12x IFB F/C 0162 0163 0167 0168 0169 0170 0171 Ports 2 2 2 2 2 4 2 Comments To I/O Drawers (FC4000) PSIFB Coupling (up to 150 meters) z9 PSIFB coupling to zEnterprise/z10 PSIFB Coupling up to 10 km (unrepeated)* To PCIe I/O Drawers (FC4003) PSIFB Coupling - up to 10 km (unrepeated)* PSIFB Coupling 150 meters
*The 1X links support a distance of up to 10 km without repeaters, RPQ 8P2263 or 8P2340 is required for 20 km support. Extended distances of up to 100 km are also possible using Dense Wavelength Division Multiplexors (DWDMs), going over 100 km requires RPQ 8P2263 or 8P2340
HCA3-O can communicate with the z10 via HCA2-O HCA3-O can not communicate with the z9 HCA2 on zEnterprise can communicate with z9 HCA1
ibm.com
yourdotcom
z196 and z114 Connectivity for Coupling and I/O - Summary

IFB IFB
HCA2-C
IFB IFB
HCA2-O
Up to 16 CHPIDs across 2 ports
IFB
HCA2-O LR
IFB
PCIe
PCIe fanout
PCIe IFB & IFB3
IFB & IFB3
HCA3-O
Up to 16 CHPIDs across 2 ports*
IFB
IFB
HCA3-O LR
IFB
IFB
Fanout cards Two-port InfiniBand host channel adapters dedicated to function HCA2-C fanout I/O Interconnect Supports FICON, ESCON, OSA, ISC-3 and Crypto Express3 cards in I/O drawer and I/O cage domains. Always plugged in pairs. HCA2-O fanout 12x InfiniBand coupling links CHPID type CIB for Coupling Fiber optic external coupling link 150 m HCA2-O LR fanout 1x InfiniBand coupling links Long Reach (carrry forward only for z114) CHPID type CIB for Coupling Fiber optic external coupling link 10 km (Unrepeated), 100 km repeated PCIe fanout I/O Interconnect Supports FICON Express8S, OSA-Express4S in PCIe drawer. Always plugged in pairs HCA3-O fanout 12x InfiniBand coupling links CHPID type CIB for Coupling Fiber optic external coupling link 150 m HCA3-O LR fanout 1x InfiniBand coupling links Long Reach CHPID type CIB for Coupling Fiber optic external coupling link 10 km (Unrepeated), 100 km repeated
* Performance considerations may reduce the number of CHPIDs per port

ibm.com
yourdotcom
z114 Drawer Combinations

All z114 Drawer Combinations I/O Drawers 0 M05 0 M10 PCIe I/O Drawers M05 1 M10 M05 2 M10 M10 M10 M10 M10 M05 M10 M05 M10 1 M05 2 M05 3 M05 4 M05
73
ibm.com
yourdotcom
IBM zEnterprise 114 (z114)

I/O Subsystem Enhancements
74
ibm.com
yourdotcom
Three subchannel sets per LCSS (z196 only)

Addition of a subchannel set of 64K devices to the existing two subchannel sets Value Extends the amount of addressable storage for the largest System z customers Provides a means to provide consistent device address definitions to help simplify addressing scheme for congruous devices Can utilize same device number in different subchannel sets The first subchannel set (SS 0) allow definitions of any type of device as is allowed today, (i.e. bases, aliases, secondaries, and those other than disk that do not implement the concept of associated aliases or secondaries) Second and third subchannel sets (SS1 and SS2) now be designated for use for disk alias devices (of both primary and secondary devices) and/or Metro Mirror secondary devices only CHPID types: ESCON - CNC, FICON - FC (both native FICON and zHPF paths) Features: ESCON, FICON Express8, FICON Express4 z/OS V1.12 z/OS V1.10 and V1.11 with PTFs
75
ibm.com
yourdotcom
IPL for alternate subchannel set

Allows to IPL from subchannel set 1 (SS1) or subchannel set 2 (SS2), in addition to subchannel set 0. Devices used early during IPL processing can now be accessed using subchannel set 1 or subchannel set 2. This is intended to allow the users of Metro Mirror (PPRC) secondary devices defined using the same device number and a new device type in an alternate subchannel set to be used for IPL, IODF, and standalone dump volumes when needed. IPL from an alternate subchannel set is supported by z/OS V1.13, as well as V1.12 and V1.11 with PTFs and applies to the FICON and zHPF protocols.
Note: SS2 for z196 only
76
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) I/O Connectivity Features
77
ibm.com
yourdotcom
z114 I/O Features supported

Supported features
I/O cage and I/O drawer Crypto Express3 (1P for z114 only) ESCON (240 or fewer) FICON Express8 (Carry forward or RPQ) FICON Express4 (Carry forward only for 4 port cards) FICON Express4-2C (Carry forward or RPQ z114) ISC-3 OSA-Express3 1000BASE-T (Includes 2P for z114) OSA-Express3 (Carry forward or RPQ) 10 GbE LR and SR, GbE LX and SX (2P for z114 only) OSA-Express2 (Carry forward only) GbE LX and SX, 1000BASE-T PSC (Carry forward or new build, no MES add) PCIe I/O drawer FICON Express8S SX and 10 km LX OSA-Express4S 10 GbE LR and SR GbE SX and LX
8 slot I/O drawer
32 slot PCIe I/O drawer
Note: PSIFB links are direct from the HCAs and not from the I/O cage or drawers
ibm.com
yourdotcom
z196 Non-Supported Features

Non-Supported Channel Types
I/O Channels FICON (before FICON Express4) FCV ESCD Model 5 Bridge Card Networking OSA-Express2 10 GbE LR OSA-Express (pre OSA-Express2) Coupling Links ICB-4 and earlier ICB Crypto Crypto Express2 and earlier ETR Sysplex Timer (ETR) Attachment
79
ibm.com
yourdotcom
FICON Express8S SX and 10KM LX in the PCIe I/O drawer

For FICON, zHPF, and FCP environments CHPID types: FC and FCP 2 PCHIDs/CHPIDs Auto-negotiates to 2, 4, or 8 Gbps
2, 4, 8 Gbps
SFP+ HBA ASIC
FLASH
IBM ASIC IBM ASIC

PCIe Switch
Increased performance compared to FICON Express8 2, 4, 8 Gbps 10KM LX - 9 micron single mode fiber Unrepeated distance - 10 kilometers (6.2 miles) Receiving device must also be LX SX - 50 or 62.5 micron multimode fiber Distance variable with link data rate and fiber type Receiving device must also be SX 2 channels of LX or SX (no mix) Small form factor pluggable (SFP) optics Concurrent repair/replace action for each SFP
SFP+
HBA ASIC
FLASH
FC 0409 10KM LX, FC 0410 SX
LX
OR
SX
80
ibm.com
yourdotcom
FICON Express8 in the I/O Drawer

Auto-negotiate to 2, 4, or 8 Gbps Connector LC Duplex 10KM LX 9 micron single mode fiber Unrepeated distance - 10 kilometers (6.2 miles) Receiving device must also be LX SX 50 or 62.5 micron multimode fiber Variable with link data rate and fiber type Receiving device must also be SX 4 channels of LX or SX (no mix)
Small Form Factor Pluggable (SFP) optics. Concurrent repair/replace action for each SFP
PCIe 2, 4, 8 Gbps PCIe 2, 4, 8 Gbps PCIe 2, 4, 8 Gbps PCIe 2, 4, 8 Gbps
FC 3325 10KM LX, FC 3326 SX
LX
OR
SX
ibm.com
yourdotcom
FICON Express8 10KM LX in the I/O Drawer

FICON Express8 Can be shared among LPARs, and defined as spanned High Performance FICON for System z (zHPF) 8 Gbps with Auto-negotiate capability (2, 4, or 8 Gbps) 1 Gbps NOT supported LX 9 micron single mode fiber Unrepeated distance - 10 kilometers (6.2 miles) Receiving device must also be LX Small Form Factor pluggable optics for concurrent repair/replace Personalize as: FC Native FICON Channel-To-Channel (CTC)
2, 4, 8 Gbps
2, 4, 8 Gbps
2, 4, 8 Gbps
2, 4, 8 Gbps
z/OS, z/VM, z/VSE, z/TPF, TPF, Linux on System z
FCP (Fibre Channel Protocol) Support of SCSI devices
FC 332510KM LX LC Duplex
z/VM, z/VSE, Linux on System z
82
ibm.com
yourdotcom
FICON Express8 SX in the I/O Drawer

FICON Express8
Can be shared among LPARs, and defined as spanned High Performance FICON for System z (zHPF)
2, 4, 8 Gbps
8 Gbps with Auto-negotiate capability (2, 4, or 8 Gbps)

1 Gbps NOT supported
2, 4, 8 Gbps
SX - 50 or 62.5 micron multimode fiber

Unrepeated distance (20 to 500 meters) varies with speed and fiber type Receiving device must also be SX
2, 4, 8 Gbps
Small Form Factor pluggable optics for concurrent repair/replace Personalize as:
FC Native FICON Channel-To-Channel (CTC) z/OS, z/VM, z/VSE, z/TPF, TPF, Linux on System z FCP (Fibre Channel Protocol) Support of SCSI devices z/VM, z/VSE, Linux on System z
2, 4, 8 Gbps
FC 3326 SX LC Duplex
ibm.com
yourdotcom
zEnterprise zHPF supports data transfers larger than 64 k bytes

zHPF multi-track data transfers are no longer limited to 64 k bytes Up to 256 tracks can be transferred a single operation Eliminating the 64 k byte limit is designed to allow a FICON Express8 channel to fully exploit its available bandwidth This enhancement is exclusive to z196 and z114 Designed to help provide Higher throughput for zHPF multi-track operations With lower response time Requires: FICON Express8 and 8S or FICON Express4 channel CHPID TYPE=FC definition Control unit support for zHPF z/OS operating system support
White Paper: High Performance FICON (zHPF) for System z Analysis http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101789 High Performance FICON (zHPF) for DS8000 System z Attached Analysis: AG Storage ATS Offering http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FLASH10668
84
ibm.com
100000
yourdotcom
I/O driver benchmark I/Os per second 4k block size Channel 100% utilized
FICON performance on System z

90000 80000 70000 60000 50000 40000 30000 20000 10000 0
z H P F z H P F FICON Express8
52000 92000
z H P F
FICON Express8S 77% increase
FICON Express4 and FICON FICON Express4 Express2 and 31000 FICON Express2 ESCON
1200 14000
FICON Express8
20000
FICON Express8S
23000
z10
z10
z196 z10
z196 z10
z196 z114
z196 z114 z H P F
1700 1600 1500 1400 1300 1200 1100 1000 900 800 700 600 500 400 300 200 100 0
I/O driver benchmark MegaBytes per second Full-duplex Large sequential read/write mix
z H P F FICON Express8 8 Gbps

620
z H P F FICON Express8 8 Gbps

770
1600
FICON Express8S 8 Gbps FICON Express8S 8 Gbps

620
108% increase
FICON Express4 4 Gbps

350
FICON Express4 4 Gbps

520
z10 z9
z10
z196 z10
z196 z10
z196 z114
z196 z114
85
ibm.com
yourdotcom
FCP performance on System z

100000 90000 80000 70000 60000 50000 40000 30000 20000 10000 0 15750 60000
I/Os per second Read/writes/mix 4k block size, channel 100% utilized

84000 84000
92000
FE4 4 Gbps
FE4 4 Gbps
31500
FE4 4 Gbps
FE8 8 Gbps
FE8 8 Gbps
FE8S 8 Gbps
10% increase
z9
z9
z10
z10
z196
z196 z114
1600 1400 1200 1000 800 600 400 200 0
MegaBytes per second (full-duplex) Large sequential Read/write mix
1600
FE8 8 Gbps FE4 4 Gbps

770 520
FE8S 8 Gbps
108% increase
z10
z196 z10
z196 z114
86
ibm.com
yourdotcom
Buffer credits for FICON Express8 and 8S

40 buffer credits available per channel Conforms to Fibre Channel standard flow control
Avoids overwriting the buffer 40 credits is 80 KB of buffer at the receiver Sufficient to hold the maximum data payload on the fiber for 20 km (round trip) of data. Anything less will not fill the buffer
Receiver side of port should contain sufficient buffer credits to ensure channel is fully utilized
Receiver must contain = > 4 buffer credits per km of fiber distance at 8 Gbps
Buffer credits are directly related to the distance supported

8 Gbps needs about 4 credits/km... 10 km = about 40 credits. 100 km = about 400 credits 4 Gbps needs about 2 credits/km... 10 km = about 20 credits. 100 km = about 200 credits
In the short block case, the concern is response time, not data bandwidth. Where the buffer-to-buffer credits (BtoB credits) becomes an issue is when large blocks per I/O are being transferred, so link bandwidth becomes the factor of interest. In this case, the frames will be almost always full (2K).
87
ibm.com
yourdotcom
z/OS Discovery and Auto-Configuration (zDAC) for z196/z114

Provides automatic discovery for FICON disk and tape control units Reduces level of IT skill and time required to configure new I/O devices Ensures system (host) and control unit definitions are compatible with each other Automatically discovers storage devices accessible to the system but not currently configured and proposes host definition values
For those discovered control units, explore for defined logical control units and devices Compare discovered logical control units and devices against those configured previously Add missing logical control units and devices to the configuration, proposing control unit and device numbers, and proposing paths to reach them Channel paths chosen using algorithm to minimize single points of failure
Integrated into existing System z host configuration tools (HCD & HCM) Requirements:
z114 or z196 server with FICON Express8S, FICON Express8 or FICON Express4 Switch/director attached fabric (no direct attachment) z/OS V1.12 (at least 1 LPAR for Dynamic I/O capability) First exploiter is IBM System Storage DS8700 DS8000 licensed machine code level 6.5.15.xx (bundle version 75.15.xx.xx), or later see IBM PSP buckets Suggested: FICON DCM (z/OS dynamic channel management) to help manage performance
88
ibm.com
yourdotcom
zDAC Value
Without I/O discovery and auto configuration, using HCD **
Defining the fibre channel network
Control unit address and ports specified on paper; requires many people and transcribing from paper to HCD.
With I/O discovery and autoconfiguration**

Discover the connections in the fiber channel network. HCD invokes IOS to discover I/O hardware in the current configuration based on autoconfiguration policies. Steps: 1 HCD: 5 panels for whole fibre channel network; policy definition and running auto discovery. Time: a few minutes
Steps: multiple iterative steps HCD: several panels per control unit Time: hours
Defining the control units: building the I/O configurations
Control Unit addresses and ports are specified manually one at a time with new or new based on function in HCD.
From the discovered control units, HCD retrieves and proposes control unit and device types and numbers, channel path assignments, partition access and OS device parameters. Definitions are automatically written into a specified target work IODF which is created as a copy of the active or accessed IODF. Steps: 1 HCD: 5 panels for whole fibre channel network Addresses and ports discovered and defined electronically, using best practices ad existing configurations. Errors discovered in working IODF and can be corrected before activation
Steps: multiple iterative steps HCD: 10 panels per control unit Time: hours
Error checking
Errors not found until IODF is activated or device is brought online, requiring rework. Manual error checking
** Based on IBM laboratory results, your results may vary

ibm.com
yourdotcom
FCP channels to support T10-DIF for enhanced reliability

System z Fibre Channel Protocol (FCP) has implemented support of the American National Standards Institute's (ANSI) T10 Data Integrity Field (DIF) standard. Data integrity protection fields are generated by the operating system and propagated through the storage area network (SAN). System z helps to provide added end-to-end data protection between the operating system and the storage device An extension to the standard, Data Integrity Extensions (DIX), provides checksum protection from the application layer through the host bus adapter (HBA), where cyclical redundancy checking (CRC) protection is implemented T10-DIF support by the FICON Express8S and FICON Express8 features, when defined as CHPID type FCP, is exclusive to z196 and z114. Exploitation of the T10-DIF standard requires support by the operating system and the storage device z/VM 5.4 with PTFs for guest exploitation Linux on System z distributions: IBM is working with its Linux distribution partners to include support in future Linux on System z distribution releases. .
90 90
ibm.com
yourdotcom
Storage Connectivity Options

Description FICON Express8S 10KM LX FICON Express8S SX FICON Express4-2C SX FICON Express4 10KM LX FICON Express4 SX FICON Express4-2C 4KM LX FICON Express4 4KM LX FICON Express8 10KM LX FICON Express8 SX F/C 0409 0410 3318 3321 3322 3323 3324 3325 3326 Ports 2 2 2 4 4 2 4 2 2 Available New New Carry Forward Carry Forward Carry Forward Carry Forward Carry Forward New New Order with RPQ 8P2534 if FC4000 slots are open Order with RPQ 8P2534 if FC4000 slots are open z114 Only Comments Initial orders Initial orders z114 Only
Maximum FICON features varies with mix of Drawers types and Model of the System All use LC Duplex connectors
ibm.com
yourdotcom
Fibre Channel Physical Interface (FC-PI-4) standard (Rev. 8.00)

Applies to FICON Express8 and 8S (2, 4, 8 Gbps), FICON Express4 (1, 2, 4 Gbps) FICON Express2 (1, 2 Gbps), and FICON Express (1, 2 Gbps) features CHPID types FC (FICON, zHPF, CTC) and FCP (Fibre Channel Protocol) Unrepeated distances in kilometers (km), meters (m), and feet (ft)
1 Gbps Fiber Core ( ) Light source 9 SM LX laser 9 SM LX laser 50 MM OM3 SX laser 50 MM OM2 SX laser 62.5 MM OM1 SX laser Distance * Link meters loss feet budget 10 km 6.2 miles 4 km # 2.5 miles 860 m 2822 ft 500 m 1640 ft 300 m 984 ft 7.8 dB 4.8 dB # 4.62 dB 3.85 dB 3.00 dB 2 Gbps Distance meters feet 10 km 6.2 miles 4 km # 2.5 miles 500 m 1640 ft 300 m 984 ft 150 m 492 ft * Link loss budget 7.8 dB 4.8 dB # 3.31 dB 2.62 dB 2.10 dB 4 Gbps Distance meters feet 10 km 6.2 miles 4 km # 2.5 miles 380 m 1247 ft 150 m 492 ft 70 m 230 ft * Link loss budget 7.8 dB 4.8 dB # 2.88 dB 2.06dB 1.78 dB 8 Gbps Distance meters feet 10 km 6.2 miles N/A 150 m 492 ft 50 m 164 ft 21 m 69 ft * Link loss budget 6.4 dB N/A 2.04 dB 1.68 dB 1.58 dB 10 Gbps ISLs Distance meters feet 10 km 6.2 miles N/A 300 m 984 ft 82 m 269 ft 33 m 108 ft * Link loss budget 6.0 dB N/A 2.6 dB 2.3 dB 2.4 dB
OM1 62.5 micron at 200 MHz-km, OM2 50 micron at 500 MHz-km, OM3 50 micron at 2000 MHz-km Inter-Switch Links (ISLs) is the link between two FICON directors; FICON features do not operate at 10 Gbps * The link loss budget is the channel insertion loss as defined by the standard. # This distance and dB budget applies to FICON Express4 4KM LX features 92 2011 IBM Corporation
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) Network connectivity
93
ibm.com
yourdotcom
System z OSA-Express3 and OSA-Express4S

Double density of ports compared to OSA-Express2
Reduced CHPIDs to manage Reduced I/O slots Reduced I/O cages or I/O drawers Up to 96 LAN ports versus 48
New microprocessor and hardware data router compared to OSA-Express2

Large send for IPv4 and IPv6 traffic Checksum offload Large send packet construction, inspection and routing performed in hardware
Designed to reduce the minimum round-trip networking time between z196/z114 systems (reduced latency)
Designed to improve round trip at the TCP/IP application layer
OSA-Express3 and 4S10 GbE 45% improvement compared to the OSA-Express2 10 GbE OSA-Express3 and 4S GbE 45% improvement compared to the OSA-Express2 GbE Designed to improve throughput (mixed inbound/outbound) OSA-Express3 and 4S 10 GbE 1.0 GBytes/ps @ 1492 MTU 1.1 GBytes/ps @ 8992 MTU 3-4 times the throughput of OSA-Express2 10 GbE 0.90 of Ethernet line speed sending outbound 1506-byte frames 1.25 of Ethernet line speed sending outbound 4048-byte frames
94
The above statements are based on OSA-Express3 performance measurements performed in a test environment on a System z10 EC and do not represent actual field measurements. Results may vary.
ibm.com
yourdotcom
OSA-Express4S GbE and 10 GbE fiber for the PCIe I/O drawer
10 Gigabit Ethernet (10 GbE) CHPID types: OSD, OSX Single mode (LR) or multimode (SR) fiber One port of LR or one port of SR 1 PCHID/CHPID
FPGA
PCIe
IBM ASIC
FC 0406 10 GbE LR, FC 0407 10 GbE SR
Gigabit Ethernet (GbE) CHPID types: OSD (CHPID OSN not supported) Single mode (LX) or multimode (SX) fiber Two ports of LX or two ports of SX 1 PCHID/CHPID Small form factor optics LC Duplex
FPGA
PCIe
IBM ASIC
FC 0404 GbE LX, FC 0405 GbE SX
ibm.com
yourdotcom
Open Systems Adapter performance

OSA processor becomes more efficient as throughput increases Window size TCP window determines amount of data that the sender can transmit to receiver without needing an acknowledgment from the receiver Faster and longer networks require larger windows to keep data flowing smoothly Blocking Performance is affected by the amount of data blocked together for transfer between OSA and TCP Frame size Larger frames perform better Larger frames reduce host and OSA processing costs Size of frame depends on LAN type, MTU setting, size of data sent Measure: throughput, transaction response time, server utilization Bulk data transfer and interactive transactions QDIO and jumbo frames (8992 byte MTUs) yield the highest streams
96 96
ibm.com
yourdotcom
OSA-Express4S 10 GbE performance (laboratory)

Inbound Streams 1492 Byte MTUs
1400 1200 2000 1000 MBps MBps 800 600 400 200 0 1500
Mixed Streams 1492 Byte MTUs

2500
80% increase
40% increase
1120 615
OSA-E3 OSA-E4S
1000
500
1180
OSA-E3
1680
OSA-E4S
Inbound Streams 8000 Byte MTUs

1400 1200
2000
Mixed Streams 8000 Byte MTUs

2500
70% increase
70% increase
Notes: AWM on z/OS z/OS is doing checksum 1 megabyte per second (MBps) is 1,048,576 bytes per second MBps represents payload throughput (does not count packet and frame headers)
1000 MBps
MBps
800 600 400 200 0
1500
1180 680
OSA-E3 OSA-E4S
1000
2080 1240
500
OSA-E3
OSA-E4S
97 97
Maximum transmission units (MTUs) 2011 IBM Corporation
ibm.com
yourdotcom
OSA-Express4S - CHPID type controls operation
CHPID type OSD All OSA features z196, z114 z10, z9, zSeries OSX 10 GbE z196, z114
Purpose / Traffic Queue Direct Input/Output (QDIO) architecture TCP/IP traffic when Layer 3 (uses IP address) Protocol-independent when Layer 2 (uses MAC address)
Operating Systems z/OS, z/VM z/VSE, z/TPF Linux on System z*
OSA-Express for zBX. Connectivity and access control to intraensemble data network (IEDN) from z196 or z114 to zBX
z/OS, z/VM
*IBM is working with its Linux distribution partners to include support in future Linux on System z distribution releases.
98
ibm.com
yourdotcom
OSA-Express3 GbE and 10 GbE fiber for the I/O Drawer

Double the port density of OSA-Express2 Reduced latency & improved throughput Ethernet hardware data router Improved throughput standard & jumbo frames New microprocessor New PCIe adapter CHPID types 10 Gigabit Ethernet OSD, OSX Gigabit Ethernet OSD and OSN
CHPID type OSN is for LPAR-to-LPAR communication. Does not use ports
PCIe
PCIe
FC 3370 10 GbE LR FC 3371 10 GbE SR
PCIe
Microprocessor PCI bus
OSA-Express2 500 MHz 10 GbE 448 MHz 1 GbE PCI-X
OSA-Express3 667 MHz PCIe G1
PCIe CHPID shared by two ports

4 ports - FC 3362 GbE LX, FC 3363 GbE SX 2 ports FC 3373 GbE SX for z10 BC and z114 only
99
ibm.com
yourdotcom
OSA-Express3 1000BASE-T copper for I/O Drawer

Auto-negotiation to 10, 100, 1000 Mbps Double the port density of OSA-Express2 Reduced latency & improved throughput Ethernet hardware data router Improved throughput standard & jumbo frames New microprocessor New PCIe adapter Usage of ports OSC, OSD, OSE can exploit four ports OSM, Port 0 only CHPID type OSN is for LPAR-to-LPAR communication. Does not use ports
Mode OSA-ICC QDIO Non-QDIO OSA for NCP (LP-to-LP) Unified Resource Manager
100
PCIe PCIe
CHPID shared by two ports
4 ports - FC 3367 1000BASE-T
2 ports - FC 3369 for z10 BC and z114 only

Microprocessor PCI bus OSA-Express2 448 MHz PCI-X OSA-Express3 667 MHz PCIe G1
CHPID OSC OSD OSE OSN OSM
Description TN3270E, non-SNA DFT, IPL CPCs, and LPARs, OS system console operations TCP/IP traffic when Layer 3, Protocol-independent when Layer 2 TCP/IP and/or SNA/APPN/HPR traffic NCPs running under IBM Communication Controller for Linux (CDLC) Connectivity to intranode management network (INMN) from z196/z114 to zBX
ibm.com
yourdotcom
OSA solutions how do they measure up?

Want to compare? Use IBM Application Workload Modeler (AWM) Replaces Netmarks, and is now externally available For z/OS and Linux on System z Can model your current network configuration in your environment Can help you allocate existing system resources more efficiently by modeling, generating real traffic on your network, and evaluating the network performance of existing workloads Rerun the same workload against a proposed alternative (e.g. OSA gigabit) Access information at http://www.ibm.com/software/network/awm/index.html
101
ibm.com
yourdotcom
Open Systems Adapter CHPIDs types: OSM (Express3 1000BASE-T)) and OSX (Express3 and 4S 10 GbE)
Two new OSA CHPID types to support new types of z196/z114 networks A z196/z114 System can have up to 6 types of OSA CHPIDs
External (customer managed) networks Defined as OSC,OSD,OSE, & OSN Existing customer provided and managed OSA ports used for access to the current customer external networks - no changes Intranode management network (INMN) Defined as CHPID type OSM, OSA-Express for Unified Resource Manager When the PCIe adaptor on 1000BASE-T is defined as CHPID type OSM, the second port cannot be used for anything else OSA-Express3 1000BASE-T configured as CHPID type OSM for connectivity to INMN from z196/z114 to Unified Resource Manager functions OSA connection via the Bulk Power Hub (BPH) on the z196/z114 to the Top of the Rack (TORs) switches on zBX Intraensemble data networks (IEDN) Defined as CHPID OSX, OSA-Express for zBX OSA-Express3 or 4S 10 GbE configured as CHPID type OSX for connectivity and access control to IEDN from z196/z114 to zBX
Functions Supported:
Dynamic I/O support HCD CP Query capabilities Ensemble Management for these new channel paths and their related subchannels.
z/VM 5.4 CHPID types OSX and OSM cannot be varied online
ibm.com
yourdotcom
CHPID Types OSX and OSM Usage
OSM (INMN)
OSA-Express3 1GbE 2 CHPIDs 2 PORTS/CHPID FC3367
OSX (IEDN)
OSA-Express3 10 GbE 2 CHPIDs 1 PORT/CHPID
IEDN Distances
MM (Short Range Optics) 50 micron at 2000 MHz-km: 300 meters (984) 50 micron at 500 MHz-km: 82 meters (269) 62.5 micron at 200 MHz-km: 33 meters (108) SM (Long Range Optics) 10 km (6.2 miles)
CHPID PORT 0
OSA-Express4S 10 GbE 1 CHPID 1 PORT/CHPID
FC3370 and FC0406 (LR) SM 9 micron LC duplex CAT 6 ETH CABLE
CHPID PORT 0
MUST USE CHPID PORT 0
FC3371 and FC0407 (SR) MM 50/62.5 micron LC duplex
Supports IOCP CHPID types: OSC, OSD, OSE, OSN, and OSM (ONLY 1000BASE-T). PCHID = xx0 & xx1
OSM IOCDS EXAMPLE:
CHPID PCHID=191,PATH=(CSS(0,1,2,3),23),TYPE=OSM,,SHARED, CNTLUNIT CUNUMBR=0910,PATH=((CSS(0),23)),UNIT=OSM IODEVICE ADDRESS=(0910,15),CUNUMBR=(0910),UNIT=OSA,UNITADD=00, MODEL=M,DYNAMIC=YES,LOCANY=YES
Supports IOCP CHPID types: OSD and OSX (ONLY 10 GbE). PCHID = xx0 & xx1
OSX IOCDS EXAMPLE:
* CHPID PCHID=5E1,PATH=(CSS(0,1,2,3),2F),TYPE=OSX,SHARED, . CNTLUNIT CUNUMBR=09F0,PATH=((CSS(0),2F)),UNIT=OSX IODEVICE ADDRESS=(09F0,15),CUNUMBR=(09F0),UNIT=OSA,UNITADD=00, MODEL=X,DYNAMIC=YES,LOCANY=YES *
103
ibm.com
yourdotcom
z196/z114 HiperSockets doubled the number

High-speed intraserver network Independent, integrated, virtual LANs Communication path system memory Communication across LPARs Single LPAR - connect up to 32 HiperSockets Support for multiple LCSS's & spanned channels Virtual LAN (IEEE 802.1q) support HiperSockets Network Concentrator Broadcast support for IPv4 packets IPv6 HiperSockets Network Traffic Analyzer (HS NTA) No physical cabling or external connections required L2 z/VM LP5 L3 z/VM LP4 z/OS LP1 Linux LP2
HiperSockets
z/VSE LP3
104
ibm.com
yourdotcom
Inbound Workload Queuing for Enterprise Extender

For improved scalability and performance Differentiate and separate inbound EE traffic to a new input queue Each input queue represents a unique type of workload Unique service and processing requirements z/OS makes decision on processing resources required Avoid resource contention Off the wire network traffic separation OSA-Express4S and OSA-Express3 (CHPID types OSD and OSX) z196 and z114 z/OS V1.13. z/VM V5.4 with PTFs for guest exploitation
105 105
ibm.com
yourdotcom
Inbound Workload Queuing for Enterprise Extender

OSA separates inbound network traffic to multiple input queues (based on type of workload) on the same host (device) interface Each input queue can be serviced concurrently by separate processors Stack receives pre-sorted packets and host processing is optimized based on the unique requirements for each unique type of workload New Enterprise Extender traffic is added as a new input queue (new IWQ)!
Other Q1 z/OS Instance (TCP/IP stack) Bulk Q2 SD Q3 EE Q4
CPU
CPU
CPU Device IWQ
CPU
EE traffic separation optimizes the host inbound EE processing
OSA-Express
WAN
106 106
ibm.com
yourdotcom
IPv6 Support
Checksum offload for IPv6 packets is now available for z/OS environments: When the checksum function is offloaded from the host, CPU cycles are reduced, improving performance. With the introduction of OSA-Express4S, the checksum offload function is now performed for IPv6 packets as well as IPv4 packets, whether the traffic goes out to the local area network (LAN), comes in from the LAN, or flows logical partition-to-logical partition through OSAExpress4S. Checksum offload provides the capability of calculating the Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Protocol (IP) header checksums for Internet Protocol Version 4 (IPv4) packets and now, IPv6 packets. Checksum verifies the correctness of files. When checksum offload was introduced in May of 2003, it was limited to IPv4 packets. Checksum offload for IPv6 packets is exclusive to OSA-Express4S features (CHPID types OSD and OSX) on the z196 and z114. It is supported by z/OS. Refer to the Software requirements section. Checksum offload for IPv4 packets is currently available for all in-service releases of z/OS and Linux on System z. Checksum offload for LPAR-to-LPAR traffic in the z/OS environment is included in the OSAExpress4S design for both IPv4 and IPv6 packets. Large send for IPv6 packets is now available for z/OS environments: Large send (also referred to as TCP segmentation offload) is designed to improve performance by offloading outbound TCP segmentation processing from the host to an OSA-Express4S feature by employing a more efficient memory transfer into OSA-Express4S. Note: Large Send for IPv6 packets is not supported for LPAR-to-LPAR packets. Large send support for IPv6 packets applies to the OSA-Express4S features (CHPID type OSD and OSX), and is exclusive to the z196 and z114. It is supported by z/OS and z/VM for guest exploitation Large send for IPv4 packets is currently available for all in-service releases of z/OS and Linux on System z.
ibm.com
yourdotcom
Open Systems Adapter in the PCIe I/O Drawer (FC4003)
Description OSA-Express4S GbE LX OSA-Express4S GbE SX OSA-Express4S 10 GbE LR OSA-Express4S 10 GbE SR
F/C 0404 0405 0406 0407
Ports 2 2 1 1
Available New Build New Build New Build New Build
CHPID OSD OSD OSD, OSX OSD, OSX
Note: OSA-Express3 1000BASE-T requires an I/O Drawer (FC4000)
108
ibm.com
yourdotcom
Open Systems Adapter in the I/O Drawer (FC4000)

Description
OSA-Express OSA-Express2 GbE LX OSA-Express2 GbE SX OSA-Express2 1000BASE-T OSA-Express2 10 GbE LR OSA-Express3 GbE LX OSA-Express3 GbE SX OSA-Express3 1000BASE-T OSA-Express3-2P 1000BASE-T OSA-Express3 10 GbE LR OSA-Express3 10 GbE SR OSA-Express3-2P GbE SX
1
F/C
23xx/13xx 3364 3365 3366 3368 3362 3363 3367 3369 3370 3371 3373
Ports
2 2 2 2 1 41 41 41 21 2 2 21
Available
NO Carry Forward Carry Forward Carry Forward NO Carry Forward Carry Forward New/Carry Forward New/Carry Forward Carry Forward Carry Forward Carry Forward
CHPID
OSD, OSN OSD, OSN OSD, OSE, OSC, OSN
OSD, OSN OSD, OSN OSD, OSE, OSC, OSN, OSM OSD, OSE, OSC, OSN OSD, OSX OSD, OSX OSD, OSN
two ports per CHPID
OSA-Express3 cards will be shown as selections via 8P2534 only if FC4000 slots are open.
ibm.com
yourdotcom
Summary: OSA-Express CHPID types to control operation

CHPID type
OSD All OSA features z196, z114, z10, z9, zSeries OSE 1000BASE-T z196, z114, z10, z9, zSeries OSC 1000BASE-T z196, z114, z10, z9, z990, z890 OSM 1000BASE-T z196 and z114 OSN GbE, 1000BASE-T z196, z114, z10, z9 No OSN support for OSA-Express4S GbE OSX 10 GbE z196 and z114
Purpose / Traffic
Supports Queue Direct Input/Output (QDIO) architecture TCP/IP traffic when Layer 3 (uses IP address) Protocol-independent when Layer 2 (uses MAC address) Non-QDIO; for SNA/APPN/HPR traffic and TCP/IP passthru traffic OSA-Integrated Console Controller (OSA-ICC) Supports TN3270E, non-SNA DFT to IPL CPCs & LPs OSA-Express for Unified Resource Manager Connectivity to intranode management network (INMN) from z196 or z114 to Unified Resource Manager functions OSA-Express for NCP Appears to OS as a device supporting CDLC protocol Enables Network Control Program (NCP) channel-related functions Provides LP-to-LP connectivity OS to IBM Communication Controller for Linux (CCL) OSA-Express for zBX Connectivity and access control to intraensemble data network (IEDN) from z196 or z114 to zBX
Operating Systems
z/OS, z/VM z/VSE, z/TPF Linux on System z z/OS, z/VM z/VSE z/OS, z/VM z/VSE z/OS, z/VM Linux on System z z/OS, z/VM z/VSE, z/TPF Linux on System z
z/OS, z/VM Linux on System z
110
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) Coupling Links Connectivity and STP
111
ibm.com
yourdotcom
ISC-3 coupling links

InterSystem Channel-3 (ISC-3) ISC-3 links ordered in increments of one Activated links balanced across features Peer mode only 2 Gbps FC 0217 (ISC-M), FC 0218 (ISC-D / ISC link) Activate link FC 0219 Four links per ISC-M Two links per ISC-D Supports 9 single mode fiber Up to 48 links per machine
ISC-M (hot-plug)
ISC-D (hot-plug)
Distances: ISC-3 Maximum distance is 10 km, RPQ 8P2197 is required for 20 km support Extended distances of up to 100 km are possible using Dense Wavelength Division Multiplexors (DWDMs) Over 100 km requires RPQ 8P2263 or 8P2340 The z114 and z196 will be the last servers to offer ordering of ISC-3. Customers should start migrating to PSIFB coupling links.
EP
ISC link
LC Duplex connector
ibm.com
yourdotcom
Parallel Sysplex using InfiniBand (PSIFB) ready for even the most demanding data sharing workloads
Simplify Parallel Sysplex connectivity Do more with less Can share physical links by defining multiple logical links (CHPIDs) Can consolidate multiple legacy links (ISC and/or ICB) Can more easily address link constraints Define another CHPID to increase available subchannels instead of having to add physical links More flexible placement of systems in a data center 12x InfiniBand coupling links (FC 0171 HCA3-O and FC 163 HCA2-O) Support optical cables up to 150 meters. No longer restricted to 7 meters between System z CPCs 1x InfiniBand coupling links (FC 0170 HCA3-O LR and FC FC 0168 HCA2-O LR) Use the same single mode fiber optic cables as ISC-3 and FICON/FCP for unrepeated distances of up to 10 km, and metropolitan distances with qualified DWDM solutions
113
ibm.com
yourdotcom
HCA3 for Coupling Links

New 12x InfiniBand and 1x InfiniBand fanout cards Exclusive to z196 and z114 HCA3-O fanout for 12x InfiniBand coupling links CHPID type CIB Improved service times with 12x IFB3 protocol Two ports per feature Fiber optic cabling 150 meters Supports connectivity to HCA2-O Link data rate of 6 GBps HCA3-O LR fanout for 1x InfiniBand coupling links CHPID type CIB Four ports per feature Fiber optic cabling 10 km without repeaters, RPQ 8P2263 or 8P2340 is required for 20 km support Extended distances of up to 100 km are also possible using Dense Wavelength Division Multiplexors (DWDMs) Over 100 km requires RPQ 8P2263 or 8P2340 Supports connectivity to HCA2-O LR Link data rate server-to-server 5 Gbps Link data rate with DWDM; 2.5 or 5 Gbps * Performance considerations may reduce the number of CHPIDs per port
HCA3-O for 12x IFB & 12x IFB3
IFB
IFB
HCA3-O LR for 1x IFB
IFB
IFB
IFB
IFB
Note: The InfiniBand link data rates of 6 GBps, 3 GBps, 2.5 Gbps, or 5 Gbps do not represent the performance of the link. The actual performance is dependent upon many factors including latency through the adapters, cable lengths, and the type of workload.
114
ibm.com
yourdotcom
HCA3-O 12x Details

New 12x InfiniBand fanout cards, exclusive to z196 and z114
*Up to 16 CHPIDs across 2 ports HCA3-O HCA3-O HCA3-O fanout for 12x InfiniBand coupling links
CHPID type CIB Two ports per feature Fiber optic cabling 150 meters Supports connectivity to HCA2-O Link data rate of 6 GBps
IFB
IFB3
Two protocols (IFB & IFB3)

1. 12x IFB = HCA3-O to HCA2-O 2. 12x IFB3 = HCA3-O to HCA3-O (see below) Improved service times, 12x IFB3 service times are designed to be 40% faster than 12x IFB
HCA2-O
HCA3-O
12x IFB3 protocol activation requirements

Maximum of four CHPIDs per HCA3-O port If more than four CHIPDs are defined per port, links will run at normal 12x IFB service times IMPORTANT ! IFB3 protocol activated as long as 4 CHPIDs or less are defined. No configuration settings required.
System z10
z196
Attachment to System z9 HCA1 not supported

115
ibm.com
yourdotcom
HCA3-O LR 1x Details
New 1x InfiniBand fanout cards, exclusive to z196 and z114
HCA3-O LR fanout for 1x InfiniBand coupling links
*Up to 16 CHPIDs across 4 ports

HCA3-O LR HCA3-O LR HCA3-O LR HCA3-O LR
CHPID type CIB

Four ports per feature Fiber optic cabling Link sharing between multiple Sysplexes Dstance of up to 10 km without repeaters, RPQ 8P2263 or 8P2340 is required for 20 km support Extended distances of up to 100 km are also possible using Dense Wavelength Division Multiplexors (DWDMs), going over 100 km requires RPQ 8P2263 or 8P2340 Supports connectivity to HCA2-O LR Link data rate server-to-server 5 Gbps Link data rate with DWDM; 2.5 or 5 Gbps
DWDM 2.5 Gbps
DWDM 5 Gpbs
Server 5 Gpbs
HCA2-O LR
HCA3-O LR
HCA3-O LR
CHPID / Subchannel Options
32 subchannels per CHPID (default) Option to define 32 or 7 subchannels z114 or z196 GA2 to z114 or z196 GA2
System z10
System z9
(not supported)
z196
z196

z114TLLB116
116
ibm.com
yourdotcom
InfiniBand coupling links on z196 and z114 Summary

Type 12x InfiniBand Speed 6 or 3 GBps Distance 150 meters Fanout HCA2-O HCA3-O HCA2-O LR* HCA3-O LR Cabling 50 MM (OM3) fiber
1x InfiniBand
5 or 2.5 Gbps
10 km
9 SM fiber
IFB
HCA2-O
IFB
Ports exit from the front of a book/processor drawer with HCAs. Unlike ISC-3 links, does not use I/O card slots 12x InfiniBand z196, z114, z10, z9 DDR at 6 GBps z196/z114 and z10 SDR at 3 GBps z196/z114 & z10 to z9 z9 to z9 connection not supported 1x InfiniBand z196/z114 and z10 (not available for z9) DDR at 5 Gbps (Server to Server and with DWDM) SDR at 2.5 Gbps (with DWDM)
DDR = double date rate, SDR = single data rate * For z114 carry forward only ** Performance considerations may reduce the number of CHPIDs per port
IFB
HCA2-O LR*
IFB
IFB & IFB3
HCA3-O
IFB & IFB3
Up to 16 CHPIDs across 2 ports**
IFB
IFB
HCA3-O LR
IFB
IFB
Up to 16 CHPIDs across 4 ports**

ibm.com
yourdotcom
InfiniBand Connectivity Options
zEnterprise to zEnterprise or z10 (HCA2-O LR)
zEnterprise to zEnterprise (HCA3) or z10 (HCA2)
zEnterprise to zEnterprise (HCA3) or z10 (HCA2)
118
ibm.com
yourdotcom
12x InfiniBand coupling links - Multiple channel paths

Up to 16 CHPIDs across two ports
HCA3-O
More subchannels per physical link CF Receiver CHPIDs can share links NOT more subchannels per CHPID MIF uses same address, 7 subchannels per CHPID
7 SubChannels per channel CHPID Up to 16 channel paths across two ports ___________________ 112 subchannels across two ports
Eg.)
CHPID FF 7 subchannels CHPID FE 7 subchannels
Single 12x IFB link 14 subchannels
119
ibm.com
yourdotcom
z114 and z196 GA2 1x InfiniBand Coupling Links Multiple CHPIDs per link, 32 or 7 subchannels per CHPID (HCA2-O LR and HCA3-O LR)
Up to 16 CHPIDs using same physical links More subchannels per physical link Link sharing by different Sysplexes
HCA3-O LR
Now more subchannels per CHPID 32 subchannels per CHPID Option to define 32* or 7 subchannels z114 or z196 GA2 to z114 or z196 GA2 32* subchannels per CHPID (default) Up to 16 CHPIDs per HCA3-O LR ___________________ 512 subchannels per HCA3-O LR
For Example:
*HCD defaults to 32 subchannels.
CHPID FF 32 subchannels
Carry forward only for z114
One 1x IFB link 64 subchannels
CHPID FE 32 subchannels
ibm.com
yourdotcom

12x InfiniBand Up to 150 meters
1x InfiniBand DDR
z196/z114 Coupling Links

Fanout, not I/O slot, used for InfiniBand ICB-4 No longer supported ETR No longer supported All coupling links support STP Sysplex Coexistence z10 EC and BC and z9 EC and BC only
.. .. .. ..
Up to 10/100KM 12x InfiniBand Up to 150 meters

1x InfiniBand DDR
HCA3-O
HCA3-O LR
Up to 10/100KM
z196, z114, z10 EC, z10 BC
12x InfiniBand SDR

.. .. .. ..
Up to 150 meters
HCA2-O* HCA2-O LR*

.. .. ..
z9 EC and z9 BC
..
HCA2-O**
ISC-3
Fanouts
IFB-MP
ISC-3 ISC-3 ISC-3 ISC-3
Up to 10/100 km
HCA2-C
z196/z114
** Not supported on the z114 ** Carry forward only
I/O Drawer or I/0 Cage
z196, z114, z10 EC, z10 BC, z9 EC, z9 BC
ibm.com
yourdotcom
System z InfiniBand HCA Adapter Options Summary

Host Channel Adapter Optical Fanouts for PSIFB
z196 supports HCA3O, HCA3-O LR, HCA2-O and HCA2-O LR fanouts z114 supports HCA3-O, HCA3-O LR and HCA2-O (carried forward only) fanouts
12X InfiniBand-SDR (3 GBps)

z196/z114 to z9 Uses 12 lanes for a total link rate of 3 GBps and is a point-to-point connection, maximum length of 150 meters New 50 micron OM3 (2000 MHz-km) multimode fiber with MPO connectors
12X InfiniBand-DDR (6 GBps)

z196/z114 to z196/z114 z196/z114 to z10 Uses 12 lanes for a total link rate of 6 GBps and is a point-to-point connection, maximum length of 150 meters 50 micron OM3 (2000 MHz-km) multimode fiber with MPO connectors
1X InfiniBand-DDR LR (5 Gbps)
z196/z114 to z196/z114 z196/z114 to z10 Uses one lane for a total link rate of 5 Gbps and supports an unrepeated distance of 10 km Repeated distances at up to 100 km when attached to a Dense Wavelength Division Multiplexer (DWDM) qualified by System z Speed may be auto-negotiated if the attached DWDM is capable of operating at SDR or DDR Supports SDR at 2.5 Gbps when connected to a DWDM capable of SDR speed Supports DDR at 5 Gbps when connected to a DWDM capable of DDR speed 9 micron single mode fiber optic cables with LC Duplex connectors (same cable as ISC3)
Note: The InfiniBand link data rates of 6 GBps, 3 GBps, 2.5 Gbps, or 5 Gbps do not represent the performance of the link. The actual performance is dependent upon many factors including latency through the adapters, cable lengths, and the type of workload. With InfiniBand coupling links, while the link data rate may be higher than that of ICB (12x InfiniBand (SDR or DDR) or ISC-3 (1x InfiniBand (SDR or DDR), the service times of coupling operations are greater, and the actual throughput may be less than with ICB links or ISC-3 links.
122
ibm.com
yourdotcom
Coupling links using InfiniBand Trade Association standard

1x InfiniBand LC Duplex connector One pair of fiber (1x) one fiber for transmit and one fiber for receive
2.5 Gbps (SDR) Fiber Core ( ) (Light source) 9 SM LR laser Optical passive loss 5 Gbps (DDR) Optical passive loss
@ wavelength
Unrepeated distance
Unrepeated distance
@ 1310 nm
10 km 6.2 miles
5.66 dB
10 km 6.2 miles
5.66 dB
1x InfiniBand operating at 2.5 Gbps or 5 Gbps is exclusive to System z10 and z196/z114 servers All attachments to an outside cable plant (including public "dark fiber") are supported only through a patch panel or Wavelength Division Multiplexer (WDM) product. 10 to 100 km repeated with an InfiniBand DDR or SDR qualified DWDM. (STP qualification is also required if STP is in use.)
123
ibm.com
yourdotcom
1x InfiniBand 9/125 micrometer single mode fiber optic cabling

LC Duplex connector
Cables available from: IBM Global Technology Services (GTS) Your preferred cable provider Fiber core 9 single mode Light source LX laser @ wavelength: @ 1310 nm
Note: the fiber optic cabling is the same as used with ISC-3, FICON LX, 10 GbE LR, and GbE LX
LC Duplex harness
124
ibm.com
yourdotcom
Coupling links using InfiniBand Trade Association standard

12x Multi-fiber Push-On (MPO) connector 24-fiber cable with Duplex MPO connectors 12 fibers for transmit and 12 fibers for receive
3 GBps (SDR) Fiber Core ( ) (Light source) 50 MM (SX laser) Fiber Bandwidth @ wavelength 2000 MHz-km @ 850 nm Unrepeated distance Optical passive loss 6 GBps (DDR) Unrepeated distance Optical passive loss
150 meters 492 feet
2.06 dB
150 meters 492 feet
2.06 dB
1. 12x InfiniBand SDR links operating at 3 GBps (2.5 Gbps per lane) are used to connect a z196/z114 to a z9 or z196/z114 to z10 2. 12x InfiniBand DDR links operating at 6 GBps (5.0 Gbps per lane) are used to connect z196/z114 servers or z196/z114 to z10
125
ibm.com
yourdotcom
Supported 12x InfiniBand DDR cable lengths OM3 50/125 micrometer multimode fiber optic cabling
Cables available from: IBM Global Technology Services (GTS) Anixter www.anixter.com/ Computer Crafts Inc. www.computer-crafts.com/ Tyco www.tycoelectronics.com/ Fujikura www.fujikura.com/ Fiber core 50u multimode Light source SX laser Fiber bandwidth @ wavelength: 2000 MHz-km @ 850 nm IBM cable part numbers highly recommended
Item Description 24-fiber cable Assembly 24-fiber cable Assembly 24-fiber cable Assembly 24-fiber cable Assembly 24-fiber cable Assembly 24-fiber cable Assembly 24-fiber cable Assembly 24-fiber cable Assembly 24-fiber cable Assembly
Duplex Duplex Duplex Duplex Duplex Duplex Duplex Duplex Duplex

126
Cable IBM P/N 41V2466 15R8844 15R8845 41V2467 41V2468 41V2469 41V2470 41V2471 42V2083
Cable Length Meters 10.0 m 13.0 m 15.0 m 20.0 m 40.0 m 80.0 m 120.0 m 150.0 m Custom
Cable Length Feet 32.8 f 42.7 f 49.2 f 65.6 f 131.2 f 262.4 f 393.7 f 492.1 f N/A
Connector Type MPO - MPO MPO - MPO MPO - MPO MPO - MPO MPO - MPO MPO - MPO MPO - MPO MPO - MPO MPO - MPO
ibm.com
yourdotcom
Non-PSIFB Coupling Links

Description ISC-D ISC-D ISC-3 Link RPQ ICB-3 ICB-4 F/C 0217 0218 0219 8P2197 0993 3393 Ports N/A 1 to 2 1 to 4 2 Available New/Carry Forward New/Carry Forward New/Carry Forward New/Carry Forward Not Available Not Available Comments Mother Card ISC-D (Daughter Card) Port(s) Enabled ISC-3 20KM
The z114 is the last server to offer ordering of new ISC-3 features. ISC-3 requires an I/O Drawer.
Server Time Protocol (STP) Required
127
ibm.com
yourdotcom
System z CF Link Connectivity (Peer Mode only)

Connectivity Options z196 / z114 / z10 / z9 ISC-3 (RPQ 8P2197 20 km) z196 / z114 / z10 / z9 ISC-3 (10/100 km) z9 EC or z9 BC with 12x InfiniBand SDR (150 m) z196 / z114/ z10 1x-InfiniBand DDR (10/100 km) z196 / z114 /z10 12x InfiniBand DDR (150 m) z196/z114 ISC-3
1 Gbps
z196/z114 1x InfiniBand
N/A
z196/z114 12x InfiniBand

N/A
2 Gbps N/A N/A N/A
N/A N/A 5 Gbps N/A
N/A 3 GBps N/A 6 GBps
Theoretical maximum rates shown 9672 / z900 / z800 / z990 / z890 Connectivity to z196/z114 Not Supported z196 and z114 does not support ICB connectivity 1x InfiniBand supports single data rate (SDR) at 2.5 Gbps when connected to a DWDM capable of SDR speed and double data rate (DDR) at 5 Gbps when connected to a DWDM capable of DDR speed or when point-to-point with another z196/z114 or z10 System z9 does NOT support 1x InfiniBand DDR or SDR InfiniBand Coupling Links
Note: The InfiniBand link data rates of 6 GBps, 3 GBps, 2.5 Gbps, or 5 Gbps do not represent the performance of the link. The actual performance is dependent upon many factors including latency through the adapters, cable lengths, and the type of workload. With InfiniBand coupling links, while the link data rate may be higher than that of ICB (12x InfiniBand (SDR or DDR) or ISC-3 (1x InfiniBand (SDR or DDR), the service times of coupling operations are greater, and the actual throughput may be less than with ICB links or ISC-3 links.
128
ibm.com
yourdotcom
System z Maximum Coupling Links and CHPIDs

Server 1x IFB (HCA3-O LR) 48 M15 32* M10 32* M05 16* N/A N/A N/A N/A 12x IFB & 1x IFB 12x IFB 12x IFB3 (HCA2-O LR) (HCA2-O) (HCA3-O) 32 M15 16* M10 16* M05 8* N/A N/A N/A N/A 32 32 M15 16* M15 16* CF only M10 12 M10 16* M05 8* M05 8* CF only 32 32 E12 16* E12 16* 12 N/A N/A 12 HCA1-O 16 S08 - 12 HCA1-O 12 IC ICB-4 ICB-3 ISC-3 Max External Links 104 (1) M10 (2) M05 (3) 64 64 64 64 Max Coupling CHPIDs 128
z196
32
N/A
N/A
48
z114 z10 EC z10 BC z9 EC z9 BC
32 32 32 32 32
N/A 16 (32/RPQ) 12 16 16
N/A N/A N/A 16 16
48 48 48 48 48
128 64 64 64 64
1. A z196 M49, M66 or M80 supports a maximum 104 extended distance links (48 1x IFB and 48 ISC-3) plus 8 12x IFB links. A z196 M32 supports a maximum 96 extended distance links (48 1x IFB and 48 ISC-3) plus 4 12x IFB links*. A z196 M15 supports a maximum 72 extended distance links (24 1x IFB and 48 ISC-3) with no 12x IFB links*. 2. z114 M10 supports a maximum of 72 extended distance links (24 1x IFB and 48 ISC-3) with no 12x IFB links*. 3. z114 M05 supports a maximum of 56 extended distance links (8 1x IFB and 48 ISC-3) with no 12x IFB links*.
* Uses all available fanout slots. Allows no other I/O or coupling.

ibm.com
yourdotcom
IBM zEnterprise 114 (z114) I/O Connectivity Summary
130
ibm.com
yourdotcom
z114 I/O Connectivity

Features ESCON FICON FICON Express8S FICON Express8 FICON Express4 FICON Express4-2C ISC-3 OSA-Express OSA-Express4S 10 GbE, GbE OSA-Express3 1000BASE-T OSA-Express3-2P 1000BASE-T OSA-Express3 10 GbE, GbE OSA-Express3-2P GbE OSA-Express2** Crypto Crypto Express3 Crypto Express3-1P NB NB 8 8 16 PCIe adapters 8 PCIE adapters 2 PCIe adapters 1 PCIe adapter 2 features*** 2 features*** NB NB NB CF* CF* CF 48 16 16 16 16 16 96 ports (48 for 10 GbE) 64 ports 32 ports 64 ports (32 for 10 GbE) 32 ports 32 ports 1 (10 GbE) / 2 ports 4 ports 2 ports 2 (10 GbE) / 4 ports 2 ports 2 ports 1 feature 1 feature 1 feature 1 feature 1 feature 1 feature NB CF* CF CF* NB 64 16 16 16 12 128 channels 64 channels 64 channels 32 channels 48 links 2 channels 4 channels 4 channels 2 channels 4 links 2 channels 4 channels 4 channels 2 channels 1 link Offered Maximum # As of features NB 16 Maximum channels 240 channels Increments per feature 1 - 15 active Purchase increments 4 channels
* Can be carried forward or ordered on MES using RPQ 8P2534 ** OSA-Express2 10 GbE LR is not supported as a carry forward *** Two features initially, one thereafter
NB = New Build CF = Carry Forward
ibm.com
yourdotcom
z114 Coupling Link connectivity summary
Features
ISC-3 HCA2-O LR* (1x) HCA2-O (12x) HCA3-O LR (1x) HCA3-O (12x) * Carry forward only
Minimum # of features
0 0 0 0 0
Maximum # of features
12 4 M05** 6 M10** 4 M05** 8 M10** 4 M05** 8 M10** 4 M05** 8 M10**
Maximum connections
48 links 8 links M05 12 links M10 8 links M05 16 links M10 16 links M05 32 links M10 8 links M05 16 links M10
Increments per feature

4 links 2 links 2 links 4 links 2 links
Purchase increments
1 link 2 links 2 links 4 links 2 links
** Uses ALL the available fanout slots for PSIFB. Allows no other I/O or coupling.
132
ibm.com
yourdotcom
IBM zEnterprise Server Time Protocol Support and Enhancements
133
ibm.com
yourdotcom
Glossary for System z Server Time Protocol (STP)

Acronym
Arbiter Arbiter
Full name
Comments
Server assigned by the customer to provide additional means for the Backup Time Server to determine whether it should take over as the Current Time Server. Server assigned by the customer to take over as the Current Time Server (stratum 1 server) because of a planned or unplanned reconfiguration. The Coordinated Server Time in a CTN represents the time for the CTN. CST is determined at each server in the CTN. A network that contains a collection of servers that are time synchronized to CST.
BTS
Backup Time Server
CST CTN CTN ID CTS
Coordinated Server Time Coordinated Timing Network
Identifier that is used to indicate whether the server has Coordinated Timing Network been configured to be part of a CTN and, if so, identifies Identifier that CTN. Current Time Server Going Away Signal A server that is currently the clock source for an STP-only CTN. A reliable unambiguous signal to indicate that the CPC is about to enter a check stopped state. The server assigned by the customer to be the preferred stratum 1 server in an STP-only CTN.
PTS
134
Preferred Time Server
ibm.com
yourdotcom
Current STP Recovery in a nut shell

CANNOT have two Stratum 1 servers in timing network Backup Time Server (BTS) can take over as Current Time Server (CTS), active Stratum 1, only if either:
Preferred Time Server (PTS) can indicate it has failed BTS can unambiguously determine the PTS has failed
Coordinated Timing Network (CTN) with only 2 servers/CFs

Based on combination of: Server Offline Signal (OLS- Channel going away signal) and Console Assisted Recovery (CAR) Server Offline signal (OLS) transmitted on a channel by the server to indicate that the channel is going offline CAR initiated when OLS not received Uses HMC/SE LAN path to determine if BTS can take over as CTS
Coordinated Timing Network (CTN) with 3 or more servers/CFs

If BTS loses communication on all established paths to CTS BTS and Arbiter communicate to establish if Arbiter also has lost communication on all established paths to CTS If both BTS and Arbiter cannot communicate with CTS BTS takes over as CTS (S1)
135
ibm.com
yourdotcom
Server Time Protocol (STP) Recovery Enhancements

The new generation of host channel adapter, HCA3-O Going away signal Server on which it is running is about to enter a failed (check stopped) state.
136
ibm.com
yourdotcom
137
ibm.com
yourdotcom
138
ibm.com
yourdotcom
139
ibm.com
yourdotcom
How it works
Going Away Signal sent by the Current Time Server (CTS) in an STPonly Coordinated Timing Network (CTN) Received by the Backup Time Server (BTS) BTS can safely take over as the CTS without relying on Offline Signal (OLS) in a 2 server CTN or the Arbiter in a CTN with 3 or more servers Current recovery design is still used when going away signal is not received by BTS and for other failures
140
ibm.com
yourdotcom
Pre-reqs
This enhancement is only available if you have a HCA3-O on the CTS communicating with a HCA3-O on the BTS. The STP recovery design is still available for the cases when a Going Away Signal is not received or for other failures besides a server failure Infiniband (IFB) links using HCA3-O to HCA3-O 12x IFB to 12x IFB HCA-O LR to HCA3-O LR 1x IFB
141
ibm.com
yourdotcom
Dependencies
Dependencies on OLS and CAR removed in a 2 server CTN Going Away Signal has priority over OLS in a 2 server CTN Dependencies on BTS-->Arbiter communication removed in CTNs with 3 or more servers BTS can also use going away signal to take over as CTS for CTNs with 3 or more servers without communicating with Arbiter
142
ibm.com
yourdotcom
Improved time coordination for virtual servers in zBX

Network Time Protocol (NTP) clients running on blades and appliances in zBX synchronize their time to the NTP server provided by the Support Element (SE) every 2 to 4 hours It is important for the SEs clock to maintain good time accuracy this enhancement provides the capability for the components in zBX to maintain an approximate time accuracy of 100 milliseconds to an NTP server if they synchronize to the SEs NTP server at least once an hour The IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise (DataPower XI50z) synchronizes to the SEs BOC once an hour. The POWER7 blade and Optimizer synchronize to the SE every 4 hours
143
ibm.com
yourdotcom
SE Enhancement
An enhancement has been made to improve the time accuracy of the SEs Battery Operated Clock (BOC) by synchronizing the SEs BOC to the servers Time of Day Clock (TOD) every hour, instead of the previous synchronization which took place every 24 hours. This enhancement will allow the SEs clock to maintain a time accuracy of 100 milliseconds to an NTP server configured as the External Time Source in an STP-only CTN.
144
ibm.com
yourdotcom
SE BOC Update
At IML SE BOC used to set the CPC TOD Prior DR93, Primary SE synchronizes with CPC TOD every 24 hours since SE BOC may have a significantly larger accumulated drift in time compared to CPC TOD Could result in repeated timestamps in logs To ensure that new Primary SE running with accurate CEC time Primary SE BOC synchs with the CPR TOD Alternate SE BOC synchs with Primary SE BOC Alternate SE switches to Primary SE
145
ibm.com
yourdotcom
zBX Virtual Servers improved time
146
ibm.com
yourdotcom
NTP thresholds
Allows you to specify threshold settings in order to suppress generation of hardware and operating system messages related to changes in the NTP server stratum level or source ID. Select the stratum level threshold value to indicate the NTP server stratum level that must be reached before generating messages. Select the source ID time threshold value to indicate the amount of time that must pass before a change in the source ID generates messages. Messages issued if the source ID does not return to the original value within the specified time period. Example: If NTP server stratum level changes from 2 to 3 and back from 3 to 2, no messages will be generated If NTP server stratum level changes from 2 to 4, messages will be generated
147
ibm.com
yourdotcom
STP Server Role Assignments (3 or more servers)

New whitepaper released - WP101833 http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101833
STP server role assignments should be accounted for and included in ALL installation / migration planning activities ! Process documented in techdoc WP101833 should be followed any time a server is take down for planned or unplanned outages. Failure to follow these procedures can result in a sysplex wide outage! Server role reassignments are easy and can be changed concurrently. Roles assignments performed under the Network Configuration panel via the System (Sysplex) Time task. All server roles assignments must be done from the CPC that is assigned as the PTS.
For PTS reassignments only, the PTS assignment to another CPC must be done from the CPC that will become the new PTS.
148
ibm.com
yourdotcom
References for Additional Information

Redbooks Server Time Protocol Planning Guide, SG24-7280 http://www.redbooks.ibm.com/redpieces/abstracts/sg247280.html Server Time Protocol Implementation Guide, SG24-7281 http://www.redbooks.ibm.com/redpieces/abstracts/sg247281.html Education Introduction to Server Time Protocol (STP) Available on Resource Link at General Availability (GA) www.ibm.com/servers/resourcelink/hom03010.nsf?OpenDatabase STP Web site www.ibm.com/systems/z/pso/stp.html Systems Assurance The IBM team is required to complete a Systems Assurance Review (SAPR Guide, SA06-012) and to complete the Systems Assurance Confirmation Form via Resource Link http://w3.ibm.com/support/assure/assur30i.nsf/WebIndex/SA779 For further information on NTP and the NTP Public services project, refer to the Web sites: http://www.ntp.org http://support.ntp.org http://www.faqs.org/rfcs/rfc1305.html
149
ibm.com
yourdotcom
IBM zEnterprise Cryptography
150
ibm.com
yourdotcom
System z Crypto History

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010/11
Cryptographic Coprocessor Facility (CCF)

G3, G4, G5, G6, z900, z800
PCI Cryptographic Coprocessor (PCICC)

G5, G6, z900, z800
PCI Cryptographic Accelerator (PCICA) z990 z800/z900 PCIX Cryptographic Coprocessor z990 (PCIXCC) CP Assist for Cryptographic Functions Crypto Express2 Crypto Express3
z990
z890 z890 z890 z990/z890 z9 EC z9 EC z9 BC z9 BC z10 EC/BC z10 EC/BC
z10 EC/BC z196/z114
Cryptographic Coprocessor Facility Supports Secure key cryptographic processing PCICC Feature Supports Secure key cryptographic processing PCICA Feature Supports Clear key SSL acceleration PCIXCC Feature Supports Secure key cryptographic processing CP Assist for Cryptographic Function allows limited Clear key crypto functions from any CP/IFL NOT equivalent to CCF on older machines in function or Crypto Express2 capability Crypto Express2 Combines function and performance of PCICA and PCICC Crypto Express3 PCI-e Interface, additional processing capacity with improved RAS 151 2011 IBM Corporation
ibm.com
yourdotcom
z196/z114 Cryptographic Enhancements

Elliptic Curve Cryptography Digital Signature Algorithm (ECC), an emerging public key algorithm expected eventually to replace RSA cryptography in many applications. ECC is capable of providing digital signature functions and key agreement functions. The new CCA functions provide ECC key generation and key management and provide digital signature generation and verification functions compliance with the ECDSA method described in ANSI X9.62 "Public Key Cryptography for the Financial Services Industry: The Elliptic Curve Digital Signature Algorithm (ECDSA)". ECC uses keys that are shorter than RSA keys for equivalent strength-per-key-bit; RSA is impractical at key lengths with strength-per-key-bit equivalent to AES-192 and AES-256. So the strength-per-key-bit is substantially greater in an algorithm that uses elliptic curves. ANSI X9.8 PIN security which facilitates compliance with the processing requirements defined in the new version of the ANSI X9.8 and ISO 9564 PIN Security Standards and provides added security for transactions that require Personal Identification Numbers (PIN). Enhanced Common Cryptographic Architecture (CCA), a Common Cryptographic Architecture (CCA) key token wrapping method using Cipher Block Chaining (CBC) mode in combination with other techniques to satisfy the key bundle compliance requirements in standards including ANSI X9.24-1 and the recently published Payment Card Industry Hardware Security Module (PCI HSM) standard. Secure Keyed-Hash Message Authentication Code (HMAC), a method for computing a message authentication code using a secret key and a secure hash function. It is defined in the standard FIPS 198, "The Keyed-Hash Message Authentication Code". The new CCA functions support HMAC using SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 hash algorithms. The HMAC keys are variablelength and are securely encrypted so that their values are protected. Modulus Exponent (ME) and Chinese Remainder Theorem (CRT), RSA encryption and decryption with key lengths greater than 2048-bits and up to 4096-bits.
152
ibm.com
yourdotcom
Summary of Cryptographic Enhancements (2011)
Crypto Express3 and Crypto Express3 -1P (z114 only) AES Key Encrypting Keys (AES KEKs) and AES typed keys, in general TR-31 key wrapping method for interoperable secure key exchange Elliptic Curve Cryptography (EC-DH key agreement protocol) PIN Decimalization table protection (Rivest-Shamir-Adleman) RSA-Optimal Asymmetric Encryption Padding (OAEP) with SHA-256
153
ibm.com
yourdotcom
TKE 7.1 LIC

Support of new hardware, firmware, software Support requirements for standards Includes using stronger/newer types of encryption inside the TKE Simplification of tasks
154
ibm.com
yourdotcom
TKE 7.1 LIC features

Security New smart card reader support Ease of Use/Simplification Key Data Migration Wizard update A single Process for Loading an Entire Key Hardware Support New Operational AES Key type Users can load and manage the new AES EXPORTER, IMPORTER, and CIPHER operational keys from the TKEs crypto module notebook PIN Decimalization Table Support CCA version 4.2 for the Crypto Express3 feature includes support for 100 decimalization tables for each domain on a host cryptographic adapter.
155
ibm.com
yourdotcom
TKE 7.1 LIC features

RAS HW Crypto module Status Display Users will be able to display the current status of the HSM that is being managed Users in privileged access mode will be able to Display Active IDs on the TKE Console Dynamic PKA Master Key Change Support
Standards/ Compliance PCI HSM Standards Compliance design Configuration Profile Support
156
ibm.com
yourdotcom
Summary of Crypto Features for z196/z114

Description CPACF enablement Crypto Express3 Crypto Express3-1P TKE 7.0 LIC TKE 7.1 LIC TKE Workstation Smart Card Reader Smart Card Additional S-Cards F/C
3863 0864 0871 0860 0867 0841 0885 0887 0884
Engines
N/A 2 1 N/A N/A N/A N/A N/A N/A
Available
New/Carry Forward New/Carry Forward New z114 /Carry Forward New/Carry Forward New New/Carry Forward New/Carry Forward (7.0) Yes New/Carry Forward
Comments
No Charge DES/TDES
z10 BC and z114 only Upgrade available to TKE 7.1 Required for z196/z114 Required for z196/z114 and ships with 7.1 LIC EC to change from NXP JCOP 41 to NXP JCOP 31
10-pack. Requires TKE 7.0 LIC or higher.
Note: Crypto Express3 requires a I/O Cage (z196 only) or I/O Drawer (FC4000)
157
ibm.com
yourdotcom
System z Evaluations & Certifications

The Common Criteria program establishes an organizational and technical framework to evaluate the trustworthiness of IT Products and protection profiles
z/VM
Common Criteria z/VM 5.3 EAL 4+ for CAPP/LSPP System Integrity Statement z/VM 6.1 Under evaluation
z/OS
Linux
z/VM
Linux Linux
Linux on System z
Common Criteria SUSE SLES10 certified at EAL4+ with CAPP Red Hat EL5 EAL4+ with CAPP and LSPP OpenSSL - FIPS 140-2 Level 1 Validated CP Assist - SHA-1 validated for FIPS 180-1 DES & TDES validated for FIPS 46-3
z/OS
Common Criteria EAL4+ with CAPP and LSPP z/OS 1.7 + RACF z/OS 1.8 + RACF z/OS 1.9 + RACF z/OS 1.10 + RACF (OSPP) z/OS 1.11 + RACF (OSPP) z/OS 1.10 IPv6 Certification by JITC IdenTrust certification for z/OS PKI Services FIPS 140-2 z/OS System SSL 1.10 z/OS System SSL 1.11 z/OS ICSF PKCS#11 Services z/OS 1.11 Statement of Integrity
158
Virtualization with partitions Cryptography

System z9 EC and z9 BC System z10 EC and z10 BC
Common Criteria EAL5 with specific target of evaluation -- LPAR: Logical partitions System z196 Common Criteria EAL5+ with specific target of Evaluation -- LPAR z114 pending Crypto Express2 & Crypto Express3 Coprocessors - FIPS 140-2 level 4 Hardware Evaluation - Approved by German ZKA CP Assist - FIPS 197 (AES) - FIPS 46-3 (TDES) - FIPS 180-3 (Secure Hash) 2011 IBM Corporation
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) RAS
159
ibm.com
yourdotcom
IBM System z Overall RAS Strategy

..Continuing our RAS focus helps avoid outages
Sources of Outages - Pre z9 -Hrs/Year/SystUnscheduled Outages Scheduled Outages Planned Outages Preplanning requirements Power & Thermal Management Impact of Outage
Scheduled (CIE+Disrupt ive Pat ches + ECs) Planned - (MES + Driver Upgrades) Unscheduled (UIRA)
Prior Servers
IBM System z9
IBM System z10
zEnterprise
In c rea sed
Fo c
us o
ver
tim e
Temperature = Silicon Reliability Worst Enemy Wearout = Mechanical Components Reliability Worst Enemy.
160
ibm.com
yourdotcom
Preventing All Outages

Unscheduled Outages Advanced Memory RAIM design Enhanced Reed-Solomon code (ECC) 90B/64B Protection against Channel/DIMM failures Chip marking for fast DRAM replacements Mirrored Key cache ECC on all Store-in caches (including new EDRAM caches) Improved SER resistant latches FCP end-to-end checking at OS level Integrated TCP/IP checksum generation/checking Enhanced STP take over architecture Integrated EPO switch cover (protecting the switch during repair actions) Continued focus on Firmware
161
ibm.com
yourdotcom

Scheduled Outages Double memory data bus lane sparing (reducing repair actions) Single memory clock bus sparing Double DRAM chipkill tolerance Field Repair of interface between processor chip and cache chip and between cache chips (fabric bus) Fast bitline delete on L3/L4 cache (largest caches) Redundant (N+2) Humidity Sensors Redundant (N+2) Altimeter Sensors PCIe based I/O infrastructure with Failover capability Concurrent replace of existing and new PCIe I/O drawer Concurrent blower repair on PCIe I/O drawer Light strip to guide repairs on PCIe IO drawer
162
ibm.com
yourdotcom

Planned Outage / Preplanning Concurrent add of existing and new PCIe I/O drawer Eliminate disruption on Cryptography Adapter (level 3 and higher) patches
163
ibm.com
yourdotcom
Improving Reliability
Power and Thermal Optimization and Management Static Power Save Mode under z/OS control FICON dynamic sub-chip level mini power down Smart blower management by sensing altitude and humidity
164
ibm.com
yourdotcom
z114 Memory Overview

Ch4 RAIM Ch3 Ch2 Ch1 Ch0 MCU1
X
Layers of Memory Recovery

ECC Powerful 90B / 64B Reed Soloman code DRAM Failure Marking technology; no half sparing needed 2 DRAM can be marked Call for replacement on third DRAM Lane Failure CRC with Retry Data lane sparing CLK RAIM with lane sparing DIMM Failure (discrete components, VTT Reg.) CRC with Retry Data lane sparing CLK RAIM with lane sparing DIMM Controller ASIC Failure RAIM Recovery Channel Failure RAIM Recovery
CLK Diff
CLK Diff
C EC
DIMM
DRAM X
C R C
X X
ASIC C R C
X X
165
ibm.com
yourdotcom
IBM System z10 BC Memory Structure

Cache
16B 16B
Key Cache
Key Cache
MCU 1
MCU 2
DATA CHECK
DATA CHECK ECC
166
ibm.com
yourdotcom
z114 Industry Exclusive RAIM Memory Structure

Cache
Key Cache
Key Cache
MCU 0
MCU 1
DATA CHECK
DATA CHECK ECC RAIM Parity
Extra column provides RAIM function
167
ibm.com
yourdotcom
z114 Model M10
2 drawer configuration New Flex cable interconnect

I/O Hub PU PU I/O Hub PU PU I/O Hub PU I/O Hub PU PU
Clocked from primary drawer redundant oscillators via cable and pass-thru card Concurrent drawer add & repair not supported Limited degrade capabilities 2 dedicated IBM spares PU assignment and sparing across drawers
Secondary Drawer Primary Drawer osc osc PU PU PU PU PU I/O Hub PU PU
I/O Hub
I/O Hub
I/O Hub
168
ibm.com
yourdotcom
z114 RAS Design

..Fully Redundant I/O Subsystem of existing IO cage and drawers
processor
Fully Redundant I/O Design

I/O Hub
Failover
I/O Hub
Failover
I/O Mux I/O Mux I/O Mux I/O Mux
Alternate Path
Network adapter. Crypro Accelerator Crypto Accelerator Network adapter. I/O adapter. I/O adapter.
I/O Switch
I/O Switch
Network Switch
Network Switch
SAP / CP sparing SAP Reassignment I/O Reset and Failover I/O Mux Reset / Failover Redundant I/O Adapter Redundant Network Adapters Redundant Crypto Accelerators I/O Switched Fabric Network Switched/Router Fabric High Availability Plugging Rules Channel Initiated Retry High Data Integrity Infrastructure Type 2 Recovery I/O Alternate Path Network Alternate Path
ISC Links
Control Unit
Control Unit
End User
End User
169
ISC Links
ibm.com
yourdotcom
z114 RAS Design

..Fully Redundant I/O Subsystem of new PCIe drawer
processor
Fully Redundant I/O Design

I/O Hub
Failover
I/O Hub
Failover
I/O Mux I/O Mux I/O Mux I/O Mux
Alternate Path
Network adapter. Crypro Accelerator Crypto Accelerator Network adapter. I/O adapter. I/O adapter.
I/O Switch
I/O Switch
Network Switch
Network Switch
SAP / CP sparing SAP Reassignment I/O Reset and Failover I/O Mux Reset / Failover Redundant I/O Adapter Redundant Network Adapters Redundant Crypto Accelerators I/O Switched Fabric Network Switched/Router Fabric High Availability Plugging Rules Channel Initiated Retry High Data Integrity Infrastructure Type 2 Recovery I/O Alternate Path Network Alternate Path
ISC Links
Control Unit
Control Unit
End User
End User
170
ISC Links
ibm.com
yourdotcom
z114 PCIe IO Drawer

Front View Conceptual packaging
AMD 1
FSP-1 card
AMD 2
Drawer Technology
.... . .... . .... . ....

Classic or Native I/O Adapters PCIe Switch Cards Light Strip
Hub cards on processor drawer Back Failover is Front To Back Domain 1 Domain 0 Front
Concurrent Adapter Repair Light strip for adapter service and upgrades Concurrent Switch card Repair Concurrent DCA repair Concurrent FSP repair Concurrent Drawer Add
New Technology
Domain 3 Domain 2
Concurrent Drawer Repair Concurrent upgrade from legacy drawer to new PCIe drawer
ibm.com
yourdotcom
zEnterprise HMC Integration System z brings Mainframe Systems Management and Service to the Blade Environment
HMC
Familiar IBM BladeCenter attached through Ethernet on z196/z114 System z HMC style operation and control of blades Redundant HMC with mirroring of critical ensemble data and switchover Dynamic provisioning of blade resources Trained IBM Service personnel for service and installation. Directed repair via Primary HMC Call home on error Dynamic failure trend monitoring Firmware management and Upgrade
Ethernet
ibm.com
yourdotcom
z114 RAS
The z114 Server continues to reduce customer down time by attack all sources of outages: unscheduled outages, scheduled outages and planned outages. Power and cooling requirements were reduced while still managing reliability. Major new Memory design for preventing outages Introducing new IO drawers, IO technologies with full concurrent service Introducing System z management to the mixed computing environment Delivering Green functions and RAS together
173
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) Capacity on Demand
174
ibm.com
yourdotcom
CoD Offerings
On-line Permanent Upgrade
Permanent upgrade performed by customer (previously referred to Customer Initiated Upgrade - CIU)
Capacity Backup (CBU)

For disaster recovery Concurrently add CPs, IFLs, ICFs, zAAPs, zIIPs, SAPs Pre-paid
Capacity for Planned Event (CPE)

To replace capacity for short term lost within the enterprise due to a planned event such as a facility upgrade or system relocation Predefined capacity for a fixed period of time (3 days) Pre-paid
On/Off Capacity on Demand (On/Off CoD)

Production Capacity to satisfy periods of peak demand for computing resources Supported through software offering Capacity Provisioning Manager (CPM) Payment: Post-paid or Pre-paid by purchase of capacity tokens Post-paid with unlimited capacity usage On/Off CoD records and capacity tokens configured on Resource Link
Customer Initiated Upgrade (CIU)

Process/tool for ordering temporary and permanent upgrades via Resource Link Permanent upgrade support: Un-assignment of currently active capacity Reactivation of unassigned capacity Purchase of all PU types physically available but not already characterized Purchase of installed but not owned memory
ibm.com
yourdotcom
Basics of CoD
Capacity on Demand
Permanent Upgrade Temporary Upgrade
Replacement Capacity
Billable Capacity (On/Off CoD) Pre-paid Post-paid
CBU
CPE
Using pre-paid unassigned capacity up to the limit of the HWM No expiration Capacity - MSU % - # Engines
176
On/Off CoD with tokens No expiration Capacity - MSU % - # Engines Tokens - MSU days - Engine days
On/Off CoD 180 days expiration Capacity - MSU % - # Engines
ibm.com
yourdotcom
Capacity on Demand Provisioning Architecture

Customer defined policy or user commands Manual operations
CPM (z/OS 1.9 or higher)
HMC application Query

* Only one On/Off CoD record can be active R1*
API Activation
Enforce Terms and Conditions and physical model limitations
R7* R8*
Authorization Layer
R2* R3* R4* R5* R6*
Up to 8 records installed and active on the System and up to 200 records staged on the SE Change permanent capacity via CIU or MES order
CBU CPE On/Off CoD
Base Model
Dormant Capacity Purchased Capacity
177
ibm.com
yourdotcom
Capacity on Demand Enhancements
z10
z196/z114
Separate orders for purchase of unassigned engines On/Off CoD records must be replenished manually CoD records staged on machine deliver No On/Off CoD administrative test
Unassigned engine purchase via CIU Auto replenishment of On/Off CoD records Manufacturing install of up to 4 CoD records with system ship. This is for new z196s/z114s only On/Off CoD Administrative tests
178
ibm.com
yourdotcom
System z Capacity Resource Availability by Server

Capacity on Demand Capacity BackUp Capacity Upgrade on Demand Customer Initiated Upgrade On/Off Capacity on Demand Capacity for Planned Event Server z196, z114, z10 EC, z9 EC, z9 BC, z990, z890, z900, z800 z196, z114, z10 EC, z9 EC, z9 BC, z990, z890, z900, z800 z196, z114, z10 EC, z9 EC, z9 BC, z990, z890, z900, z800 z196, z114, z10 EC, z9 EC, z9 BC, z990, z890 z196, z114, z10 EC, z10 BC Customer Initiated Upgrade for ordering
Yes (CP, IFL, ICF, Memory) Yes (CP, IFL, ICF) Yes (CP, IFL, ICF, zAAP, Memory*) Yes (CP, IFL, ICF, zAAP, zIIP, Memory*) Yes (CP, IFL, ICF, zAAP, zIIP, SAP, Memory*)
Capacity BackUp Capacity Upgrade on Demand

Yes (CP only) Yes (CP only) Yes (CP only) Yes (CP, ICF, IFL zAAP, zIIP) Yes (CP, ICF, IFL zAAP, zIIP, SAP) Yes (CP, I/O, IFL, ICF, Memory) Yes (CP, I/O, IFL, ICF) Yes (CP, I/O, IFL, ICF, zAAP, Memory*) Yes (CP, I/O, IFL, ICF, zAAP, zIIP, Memory*) Yes (CP, I/O, IFL, ICF, zAAP, zIIP, SAP, Memory*)
On/Off Capacity on Demand

No No Yes (CP, IFL, ICF, zAAP) Yes (CP, IFL, ICF, zAAP, zIIP) Yes (CP, IFL, ICF, zAAP, zIIP, SAP)
Capacity for Planned Event

No No No No Yes (CP, IFL, ICF, zAAP, zIIP SAP)
z900 z800 z890/z990 z9 z10/z196/z114
* Not supported for some memory upgrades Note: Upgrades are non-disruptive only where there is sufficient hardware resource available and provided pre-planning has been done
179
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) Capacity and Performance Planning
180
ibm.com
yourdotcom
z114 Capacity Matrix

Z01 Y01 X01 W01 V01 U01 T01 S01 R01 Q01 P01 O01 N01 M01 L01 K01 J01 I01 H01 G01 F01 E01 D01 C01 B01 A01 1-way Specialty Engine Z02 Y02 X02 W02 V02 U02 T02 S02 R02 Q02 P02 O02 N02 M02 L02 K02 J02 I02 H02 G02 F02 E02 D02 C02 B02 A02 2-way Specialty Engine Z03 Y03 X03 W03 V03 U03 T03 S03 R03 Q03 P03 O03 N03 M03 L03 K03 J03 I03 H03 G03 F03 E03 D03 C03 B03 A03 3-way Specialty Engine Z04 Y04 X04 W04 V04 U04 T04 S04 R04 Q04 P04 O04 N04 M04 L04 K04 J04 I04 H04 G04 F04 E04 D04 C04 B04 A04 4-way Specialty Engine Z05 Y05 X05 W05 V05 U05 T05 S05 R05 Q05 P05 O05 N05 M05 L05 K05 J05 I05 H05 G05 F05 E05 D05 C05 B05 A05 5-way Specialty Engine Specialty Engine Specialty Engine Specialty Engine Specialty Engine Specialty Engine
z114
Granularity levels similar to z10 BC to facilitate upgrades and incremental growth Nomenclature: XYY X = Capacity level YY= Number of processors A00 = ICF or IFL only Any to any capacity upgrade/downgrade capability within the Model CBU capability from smallest to largest capacities within the Model On/Off CoD within the Model Linux only and ICF only servers Model M10 provides specialty engine scale out capabilities
Additional engines available on the M10
181
ibm.com
yourdotcom
z114 Sub-capacity Processor Granularity

The z114 has 26 CP capacity levels (26 x 5 = 130) Up to 5 CPs at any capacity level All CPs must be the same capacity level The one for one entitlement to purchase one zAAP and/or one zIIP for each CP purchased is the same for CPs of any speed. All specialty engines run at full speed Processor Unit Value for IFL = 100
Number of z114 CPs Base Ratio Ratio z10 BC to z114
1 CP 2 CPs 3 CPs 4 CPs 5 CPs
z10 BC Z01 z10 BC Z02 z10 BC Z03 z10 BC Z04 z10 BC Z05
1.18 1.16 1.14 1.13 1.12
5-way 3139 PCIs
1-Way
5-Way 4-Way 3-Way 2-Way
ABC
1-way (sub-capacity 26 PCIs)
DE F GH I J K L MN O P Cap Q R acit S T y le vel U V W X
#E
e gi n
1-way 782 PCIs
PCI Processor Capacity Index

FULL size Specialty Engine
ibm.com
yourdotcom
System z Capacity Planning in a nutshell
GHz
MIPs tables
HiperDispatch
MSUs ratings
Dont use one number capacity comparisons! Work with IBM technical support for capacity planning! Customers can now use zPCR
The IBM Processor Capacity Reference (zPCR) is a free tool available for download that can be used to size your System z processors. http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS1381 183 2011 IBM Corporation
ibm.com
yourdotcom
Whats Changing for z114

Single set of LSPR data, based on z/OS 1.11 Multi-Image LSPR Table (typical partition configurations assumed based on N-way)
Supports up to 5 General Purpose CPs on z114 Supports up to 80 General Purpose CPs (z196)
Single-Image LSPR Table (single partition configurations assumed)

Supports up to 5 General Purpose CPs or 10 IFLs in single image on z114 Supports up to 80 General Purpose or IFL CPs in a single image (z196) *
LSPR workloads
Three hardware-characteristic-based workload categories Low, Average, and High categories, based on Relative Nest Intensity Previous workload primitives and Mixes were replaced
Reference Processor Typical Setting

2094-701 @ 593 MIPS for a single-partition configuration Equates to 560 MIPS for a 5-partition configuration (multi-image LSPR table)
Capacity planning tools affected

zPCR, zCP3000, BWATOOL, zPSG, zTPM
* Note: Whenever considering a z/OS partition with more than 48 CPs in a single partition, send a note to zpcr@us.ibm.com for guidance.
ibm.com
yourdotcom
LSPR Tables
Multi-image (MI) Processor Capacity Ratio table Average complex LPAR configuration for each model based on customer profiles Most representative for vast majority of customers Same workload assumed in every partition z/OS only Use for high level sizing Used to develop the MSU rating Single-image (SI) Processor Capacity Ratio table * One z/OS partition equal in size to N-way of model (up to 80-way) This table is ONLY relevant for truly single image z/OS cases Includes z/VM and Linux workloads on both GP and IFL CPs *Note: Likely to be dropped in a future release of zPCR Single-Image table is used as the basis for IBM capacity planning tools such as zPCRs LPAR Configuration Capacity Planning function. Multi-Image table is a generalization of relative capacity, and is not useful for capacity planning purposes. Capacity for z/OS on z10 and later processors is represented with HiperDispatch turned ON.
ibm.com
yourdotcom
LSPR Versions Supported

LSPR Data (supporting z114) z114, and IBM System z (measured in LPAR-mode) z/OS 1.11 (Low, Average, High)* z/VM (High/LV) Linux (WASDB/L) zPCR supplements published workload categories Low-Avg Avg-High Prior LSPR Data and Tables Removed from zPCR July 2010 (z/OS 1.9, z/OS 1.8, z/OS 1.6 and z/OS 1.4) All processor data prior to System z (G6 and before) Removed from zPCR November 2009 Removed from zCP3000 May 2010 All capacity planning results will be based on the most recently published data
*Note: Measured using HiperDispatch; applies to z10, z196 and z114 only
ibm.com
yourdotcom
LPAR configuration Considerations

z/OS 1.11
Supports up to 80 CPs LSPR data for 1 through 80 CPs assumes a single partition Supports zAAP/zIIP Logical CPs Measured using HiperDispatch* Minimizes cross book effects Attempts to keep important Logical CPs on same Real CPs Improves high speed buffer efficiency
Other z/OS releases may be specified

z/OS 1.4 through z/OS 1.10 and z/OS 1.12 z/OS 1.11 LSPR capacity data will be used Enforce the SCP rules
z/VM 5.4
Supports up to a maximum of 32 CPs LSPR measured up to 32 CPs
Linux
zPCR allows up to up to a maximum of 32 CPs LSPR measured up to 16 CPs
z/VSE
zPCR allows up to a maximum of 4 CPs
CFCC
Supports up to a maximum of 16 CPs
*Note: Capacity Planning for z10 Upgrades document is available TD104738

ibm.com
yourdotcom
New LSPR Workload Categories

Various combinations of prior workload primitives are measured on which the new workload categories are based Applications include CICS, DB2, IMS, OSAM, VSAM, WebSphere, COBOL, utilities Low (relative nest intensity) Workload curve representing light use of the memory hierarchy Similar to past high scaling workload primitives Average (relative nest intensity) Workload curve expected to represent the majority of customer workloads Similar to the past LoIO-mix curve High (relative nest intensity) Workload curve representing heavy use of the memory hierarchy Similar to the past DI-mix curve zPCR extends published categories Low-Avg 50% Low and 50% Average Avg-High 50% Average and 50% High
ibm.com
yourdotcom
Choosing a Workload Category

Average is the default when no other information is available Based on DASD I/O Rate From RMF Workload Activity report Captured in EDF using CP3KEXTR* (one file per partition) Based on SMF 113 Data (alternate view currently provided as hint) Based on CPU MF counter data Captured by partition Intended to eventually provide a more reliable way to choose a workload Not yet validated likely to be modified based on feedback Captured in EDF using CP3KEXTR* (one file per partition) Not required
*Note: CP3KEXTR is a Load and Go program written in Assembler. Runs under z/OS. Available to customers since September 2010
189
ibm.com
yourdotcom
System z New Workloads Sizing

SAP Sizing for any IBM platform should be done through Techline (e.g., esizings@us.ibm.com) Can also be used if you need a z114 sizing (less accurate) before announce (under Letter 112) zPSG Sizing for new applications running on System z. Supports WebSphere (Application Server, Portal Server, Process Server, Message Broker and MQ), DB2 for z/OS Transaction sizing Engage Global Techline Solutions Sizing Support (choose your geography from the drop down) Additional support for workloads not found in zPSG Cognos for Linux on System z IBM Content Manager Domino Mail Will zPSG be available to IBM BPs via Partner World
190
ibm.com
yourdotcom
Cross Platform sizing including POWER7 blades

Size the z196 portion or z114 portion Engage a Techline specialist to help you collect the data and do the sizing via Deal Hub Connect Use zPCR or zCP3000 Use CP2KEXTR and CP3KVMXT to create an EDF file for z/OS and z/VM Complete data collection guides located here: http://w3.ibm.com/techdocs/PRS2664 - for z/OS http://w3.ibm.com/techdocs/PRS2875 - for z/VM IBM Business Partners will be able to obtain the tools directly from PartnerWorld refer to slide 24. Size the POWER7 portion allow at least one week. Currently a manual process 20-50 LPARS should take a week More complex environments would take longer Working towards a more automated process Sizing when migrating from competitive machines to POWER7 blades Engage a Techline specialist via Deal Hub Connect to help you collect the data and do the sizing Identify which machines and which time periods the customer would like to consider Collect data from the competitive machines covering the time frames Server consolidation data collection guidance located here: http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS1423 Sizing new applications running on POWER7 blades Engage Global Techline Solutions Sizing Support via Deal Hub Connect Software sizing questionnaires located here: http://w3-03.ibm.com/support/techline/sizing/tg_im_sizing.html
191
ibm.com
yourdotcom
Cross Platform sizing including POWER7 blades

Sizing when the customer has an existing set of IBM servers they would like to migrate to POWER7 blades (go ahead and collect the data now) Identify which machines and which time periods the customer would like to consider in the proposal Collect data from AIX covering the time frames
Work hand in hand with a POWER7 Specialist to collect the data and do the sizing
OR
Engage a Techline specialist via Deal Hub Connect to help you collect the data and do the sizing Sizing questionnaires located here: http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4034
We're working with the WLE (Workload Estimator) team for support Targeted to be available at announce Techline will deliver World Wide education when the support is available
IBM Business Partners PWCS (PartnerWorld Contact Services)
192
ibm.com
yourdotcom
Cross Platform sizing including Select IBM System x blades

Size the z196 portion or z114 portion Engage a Techline specialist to help you collect the data and do the sizing via Deal Hub Connect Use zPCR or zCP3000
Use CP2KEXTR and CP3KVMXT to create an EDF file for z/OS and z/VM Complete data collection guides located here: http://w3.ibm.com/techdocs/PRS2664 - for z/OS http://w3.ibm.com/techdocs/PRS2875 - for z/VM
IBM Business Partners will be able to obtain the tools directly from PartnerWorld. we're working with the WLE (Workload Estimator) team for support Targeted to be available at announce Techline will deliver World Wide education when the support is available
193
ibm.com
yourdotcom
IBM Systems Workload Estimator

Welcome to the IBM Systems Workload Estimator IBMers access: http://wletest.rchland.ibm.com/wleProd/EstimatorServlet IBM Business Partners access: http://www-912.ibm.com/wle/EstimatorServlet
194
ibm.com
yourdotcom
IBM WebSphere DataPower XI50 for zEnterprise (DataPower XI50z)

IBM DataPower XI50z can perform a tremendous variety of use cases, and in almost all real world use cases, there are other servers involved, which introduce their own latency. Currently DataPower XI50z sizing typically is performed on a per client basis, often based on results of a proof of concept. If a System z seller identifies the opportunity, they should bring in a SWG WebSphere seller who can then decide whether to bring in a SWG DataPower Tech Sales Specialist. IBM Business Partners Must be certified to sell DataPower XI50z Should already posses the skills to do the sizing
195
ibm.com
yourdotcom
IBM zEnterprise 114 (z114) Upgrades
196
ibm.com
yourdotcom
z114 Upgrade paths

z10 BC
z114
z196 Model M15
M10 upgrade path only
M10
Yes
M05 upgrade Not Supported
M05
NO
Air Cooled only
z9 BC
Upgrade paths from z9 BC and z10 BC Upgrade path to z196 Model M15 (Air cooled only) Disruptive upgrade M05 to M10 and from M10 to z196 M15
ibm.com
yourdotcom
zEnterprise 114 Product Positioning

New modular based packaging construct. Increased capacity in a single footprint Designed for up to 1.12 times the z10 BC in total system capacity out-of-order instruction processing higher clock frequency larger caches compiler enhancements New I/O subsystem infrastructure Connectivity improvements include bandwidth and throughput
z114 MIPS capacity in a single footprint
z10 BC
z9 BC
RAIM fault tolerant memory Federated capacity and hybrid computing capabilities via integration with zBX, application server blades, optimizers and Unified Resource Manager.
z890
198
PUs for Customer use

ibm.com
yourdotcom
Summary
199
ibm.com
yourdotcom
zEnterprise 114 in a nutshell:

Integration with the zEnteprise BladeExtension Two Models Increased capacity in a single footprint 12s0 technology out-of-order instruction processing higher clock frequency larger cache Robust Memory Upgrades Investment protection with upgrades from two previous families z10 BC z9 BC Upgradeability to z196 (M15) I/O Improvements new I/O features and New I/O Drawer
200
ibm.com
yourdotcom
Summary: Deploy Multi-tier workloads on a single integrated system Create an integrated infrastructure that aligns to end-to-end business processes across technology boundaries
Scale without adding complexity to meet the growing demands on the infrastructure Reduce cost through centralized data center management and automation Fit for purpose application deployment Unify and optimize multiple architectures to work as a single system Improve performance with co-location of data and applications Improve resiliency through a private network Enable innovation through flexibility and breaking down silos
Extending the reach and strategic role of the mainframe across the enterprise; simplify and reduce the range of skills necessary for managing the datacenter; break down cultural divides.
ibm.com
yourdotcom
IBM zEnterprise System
Questions ?
Parwez Hamid parwez_hamid@uk.ibm.com Luiz Fadel fadel@br.ibm.com
202
ibm.com
yourdotcom
Dank u
Dutch
Merci
French
c
Russian
Gracias
Spanish

Arabic
Korean
Tack s mycket
Swedish
Hindi

Hebrew
Obrigado
Brazilian Portuguese
Chinese
Dankon
Esperanto
Thank You
Trugarez
Breton
Japanese
Tak
Danke
German Danish
Grazie
Italian
}
Tamil
dkuji
Czech
go raibh maith agat

Gaelic
Thai
ibm.com
yourdotcom
End of Presentation
204
ibm.com
yourdotcom
I/O Cage/Drawer Migration
205
ibm.com
yourdotcom
z114 & z196 GA2 I/O Drawers and Cages

PCIe I/O drawer
z114 & z196

4 I/O domains 32 I/O slots (PCIe I/O cards only) At least two PCIe fanouts (4 ports per drawer) 7 EIA Units
FC 4003
I/O drawer
z114, z196, & System z10 BC

2 I/O domains 8 I/O slots (legacy I/O cards only) At least 2 HCA2-C fanouts (2 ports per drawer) Up to two drawers on a pair of fanouts 5 EIA Units
FC 4000
z196 & earlier (except z10 BC)

z114
Not Supported
7 I/O domains 28 I/O slots (legacy I/O cards only) Up to 4 fanouts for all 7 domains 14 EIA Units
206
ibm.com
yourdotcom
z196/z114 I/O Cage/Drawer Migration

With the introduction of the PCIe I/O drawer, a strategy has been developed to help migrate customers I/O from the current I/O cage and I/O drawer to the PCIe I/O drawer I/O cage/drawer migration goals Eliminate the need for I/O cages and I/O drawers by two generations after the z196 I/O cage/drawer migration execution I/O feature ordering I/O feature carry forward
207
ibm.com
yourdotcom
I/O Feature Ordering and Carry Forward*

Customers cannot order I/O cages (z196) or I/O drawers, they order I/O features and eConfig will determine the proper mix of I/O cage (z196), I/O drawer, and the PCIe I/O drawer New builds, migration offerings, and z exchange programs All FICON and OSA features (except 1000 BASE-T) will be PCIe I/O based Crypto Express3, ESCON, ISC-3, OSA-Express3 1000BASE-T and PSC will be I/O drawer based Box MES and loose piece MES RPQ to order the I/O drawer based cards to fill empty I/O cage and I/O drawer slots until they are filled
RPQ is only available if there are no open PCIe I/O drawer slots Current I/O drawer based cards are FICON Express8 and OSA-Express3, Crypto Express3
and ISC-3 coupling links Once the empty slots (i.e. I/O cage & I/O drawer) are filled, eConfig will drive out a PCIe I/O drawer with the associated features
Customers will not be allowed to carry forward the oldest generation of I/O cards in future severs* z114 will not support (i.e. allow the carry forward) of FICON Express4, OSA-Express2 and ESCON features Future servers will not support FICON Express8, OSA-Express3, ISC-3, or PSC features
*Refer to the Statements of Direction

ibm.com
yourdotcom
ESCON Statement of Direction* - February 15, 2011

The IBM zEnterprise 196 (z196) will be the last high-end server to support ESCON channels: IBM plans not to offer ESCON channels as an orderable feature on high-end System z servers which follow the z196 (machine type 2817). In addition, ESCON channels cannot be carried forward on an upgrade to such a follow-on server. This plan applies to channel path identifier (CHPID) types CNC, CTC, CVC, and CBY and to feature code numbers 2323 and 2324. System z customers should continue migrating from ESCON to FICON. Alternate solutions are available for connectivity to ESCON devices. IBM Global Technology Services offers an ESCON to FICON Migration solution, This offering should help customers to simplify and manage a single physical and operational environment. Notes:
This new Statement of Direction supersedes the previous ESCON SOD in Announcement letter 110170 of July 22, 2010. It also confirms the SOD in Announcement letter 109-230 of April 28, 2009 that ESCON Channels will be phased out. This SOD does NOT say that the z10 BC will be the last midrange server to support ESCON channels or the last to offer ESCON channels as an orderable feature.
*All statements regarding IBM's plans, directions, and intent are subject to change or withdrawal without notice. Any reliance on these statements of general direction is at the relying party's sole risk and will not create liability or obligation for IBM.
ibm.com
yourdotcom
FICON Convertor Support
Partnership
Old
http://www-935.ibm.com/services/us/index.wss/itservice/igs/a1026000?cm_re=masthead-_-itservices-_-site The order process for Prizm is the same as it is for IBM cabling systems 210 2011 IBM Corporation
Current
Prizm is available from IBM GTS Site & Facilities as part of the EFM Service (ESCON to FICON Migration - offering # 6948-97D)
ibm.com
yourdotcom
http://www.opticatech.com/ To help reduce or eliminate ESCON channels on the host while maintaining ESCON and Bus/Tag-based devices and applications
Optica Technologies PRIZM FICON Converter
What is PRIZM?
A purpose-built appliance designed exclusively for IBM System z to enable a smooth transition from use of ESCON channels on the host to use of FICON channels on the host Allows ESCON devices to connect to FICON channels and FICON fabrics/networks Also designed to support attachment of parallel (bus/tag) devices to FICON channels via ESBT module Converts 1 or 2 FICON channels into 4, 8 or 12 ESCON channels Replace aging ESCON Directors with PRIZM Achieve streamlined infrastructure and reduced Total Cost of Ownership Qualified by the IBM Poughkeepsie, NY Vendor Solutions Lab in July 2009 for all ESCON devices Refer to: http://www-03.ibm.com/systems/z/hardware/connectivity/index.html Products -- > FICON / FCP Connectivity -- > Other supported devices PRIZM is available via IBM Global Technology Services: ESCON to FICON Migration offering (#6948-97D) Front
FICON Ports: LC Duplex
Back
ESCON Ports: MT-RJ
211
ibm.com
yourdotcom
Where Does PRISM Fit in the Data Center?

Topologies supported by PRISM
Local: direct attached or switched Remote: ISL (cascaded) or IP channel extended
FICON Tape FICON
Prizm supports local OR remote FICON attachment via ISL or IP WAN
FICON Disk
FICON
FICON
ESCON Tape ESCON
ESCON Tape
ESBT Module
B/T
Parallel (B/T) Printers
212
ibm.com
yourdotcom
Additional Material
213 213
ibm.com
yourdotcom
PCHID Ranges I/O Card Slot LG01 Drawer Type I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer
A02B
A09B DCA 1
A16B
A26B (M05 only)
100 - 103 100 - 10F 104 - 107 110 - 11F 108 - 10B 120 - 12F 10C - 10F 130 - 13F
180 - 183 180 - 18F 184 - 187 190 - 19F 188 - 18B 1A0 - 1AF 18C - 18F 1B0 - 1BF
X 200 - 20F X 210 - 21F X 220 - 22F X 230 - 23F X DCA 2
X 280 - 28F X 290 - 29F X 2A0 - 2AF X 2B0 - 2BF X
LG02
LG03
LG04
LG05
PCIe Switch 0
LG06
110 - 113 140 - 14F 114 - 117 150 - 15F 118 - 11B
190 - 193 1C0 - 1CF 194 - 197 1D0 - 1DF 198 - 19B IFB MP
X 240 - 24F X 250 - 25F X
X 2C0 - 2CF X 2D0 - 2DF X
LG07
LG08
LG09
11C - 11F 160 - 16F
19C - 19F 1E0 - 1EF
X 260 - 26F X
X 2E0 - 2EF X
LG10
214
FSP-1,1
ibm.com
yourdotcom
PCHID Ranges I/O Card Slot LG11 Drawer Type I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer
A02B 170 - 17F 120 - 123 X 124 - 127 X 128 - 12B X 12C - 12F X
A09B 1F0 - 1FF 1A0 - 1A3 X 1A4 - 1A7 X 1A8 - 1AB X 1AC - 1AF X
A16B 270 - 27F X X X X X X X X X X X X X X X X X X X
A26B (M05 only) 2F0 - 2FF X X X X X X X X X X X X X X X X X X X
LG12
LG13
LG14
LG15
PCIe Switch 2 X 130 - 133 X 134 - 137 X 138 - 13B X 13C - 13F X 140 - 143 X 1B0 - 1B3 X 1B4 - 1B7 X 1B8 - 1BB X 1BC - 1BF X 1C0 - 1C3
LG16
LG17
LG18
LG19
LG20
215
PCIe I/O Drawer
ibm.com
yourdotcom
PCHID Ranges I/O Card Slot LG21 Drawer Type I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer
A02B X 144 - 147 X 148 - 14B X 14C - 14F X
A09B X 1C4 - 1C7 X 1C8 - 1CB X 1CC - 1CF X
A16B X X X X X X X X X X X X X X X X X X
A26B (M05 only) X X X X X X X X X X X X X X X X X X X X
LG22
LG23
LG24
PCIe Switch 3 X 150 - 153 X 154 - 157 X 158 - 15B X 15C - 15F X FSP-1,2 X 160 - 163 X 1E0 - 1E3 X 1D0 - 1D3 X 1D4 - 1D7 X 1D8 - 1DB X 1DC - 1DF X
LG25
LG26
LG27
LG28
LG29
LG30
216
X X
ibm.com
yourdotcom
PCHID Ranges I/O Card Slot LG31 Drawer Type I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer I/O Drawer PCIe I/O Drawer A02B X 164 - 167 X 168 - 16B X 16C - 16F X A09B X 1E4 - 1E7 X 1E8 - 1EB X 1EC - 1EF X A16B X X X X X X X X X X X X X X X X A26B (M05 only) X X X X X X X X X X X X X X X X
LG32
LG33
LG34
PCIe Switch 1 X 170 - 173 X 174 - 177 X 178 - 17B X 17C - 17F X 1F0 - 1F3 X 1F4 - 1F7 X 1F8 - 1FB X 1FC - 1FF
LG35
LG36
LG37
LG38
217

2 Zenterprise Technical Details - ITSOWW2011

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

2 Zenterprise Technical Details - ITSOWW2011

Încărcat de

Drepturi de autor:

Formate disponibile

yourdotcom

International Technical Support Organization and Authoring Services

zEnterprise Technical Details

Parwez Hamid Executive IT Consultant IBM UK Limited

2011 IBM Corporation

International Technical Support Organization and Authoring Services

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

The following are trademarks or registered trademarks of other companies.

2011 IBM Corporation

International Technical Support Organization and Authoring Services

zEnterprise Technical Details z114

International Technical Support Organization and Authoring Services

IBM System z Family

2011 IBM Corporation

International Technical Support Organization and Authoring Services

IBM System z10 family

Memory up to 1.5 TB for Server and up to 1 TB per LPAR

Memory up to 256 GB for Server

2011 IBM Corporation

International Technical Support Organization and Authoring Services

IBM zEnterprise family

Announced 07/11 2 models M05 and M10

High levels of Granularity available

Memory up to 3 TB for Server and up to 1 TB per LPAR

Memory up to 256 GB for Server

2011 IBM Corporation

International Technical Support Organization and Authoring Services

z114 continues the CMOS Mainframe heritage

1.4 GHz 1.0 GHz

625 MHz 139 MHz 413 MHz

2011 IBM Corporation

International Technical Support Organization and Authoring Services

IBM zEnterprise 114 (z114) Models, Frame and Processor Drawers

2011 IBM Corporation

International Technical Support Organization and Authoring Services

2011 IBM Corporation

International Technical Support Organization and Authoring Services

zEnterprise 114 Models M05 and M10

International Technical Support Organization and Authoring Services

zEnterprise 114 Functions and Features (GA Driver 93 September, 2011)

Doubled HiperSockets to 32 Additional STP enhancements Doubled Coupling CHPIDs to 128

International Technical Support Organization and Authoring Services

zEnterprise zBX Functions and Features

Advance Management Module 1000BASE-T and 10 GbE TORs

2011 IBM Corporation

International Technical Support Organization and Authoring Services

z114 Model M10 Under the covers

Service Processor (FSP) cage controller

PCIe I/O drawers

FICON & ESCON FQC

International Technical Support Organization and Authoring Services

CEC Drawers Placement

2011 IBM Corporation

International Technical Support Organization and Authoring Services

Processor / Memory Subsystem Drawers (Model M05 and M10)

International Technical Support Organization and Authoring Services

z114 Processor drawer packaging

2 x I/O Fanouts Drw to Drw Interconnect

2011 IBM Corporation

International Technical Support Organization and Authoring Services

z114 Processor drawers Infrastructure (M10)

International Technical Support Organization and Authoring Services

Oscillator / Fabric Flex Cable