IBM Power Systems Technical University featuring IBM AIX and Linux

September 8 12, 2008 Chicago, IL

Session Title: Configuration of Virtual I/O on POWER6 Session ID: pVI08

Speaker Name: Janel Barfield
IBM UNIX Technical Support Education email: jgbarfie@us.ibm.com
2008 IBM Corporation

IBM Training

Agenda
Virtual I/O configuration concepts Describe and configure virtual Ethernet Describe the new Integrated Virtual Ethernet (IVE) adapter
Architecture Multi-Core Scaling

Configure external network access using IVE Describe and configure external network access using Shared Ethernet adapter (SEA)
With Link Aggregation With SEA Failover

Describe and configure virtual SCSI

Configure new file-backed virtual devices

Conclusion
2008 IBM Corporation

IBM Training

Virtual I/O Configuration Concepts

On POWER5 and POWER6, virtual I/O devices include virtual Ethernet, Integrated Virtual Ethernet Adapter, Shared Ethernet Adapter, and virtual SCSI devices Integrated Virtual Ethernet is a new POWER6 feature Virtual SCSI and Shared Ethernet adapters require the Advanced POWER Virtualization feature and a configured Virtual I/O Server The Virtual I/O Server is a dedicated special appliance partition

2008 IBM Corporation

IBM Training

Virtual Ethernet: POWER5 Hypervisor Ethernet Switch

The POWER Hypervisor provides the Ethernet switch functionality for virtual Ethernet adapters on POWER5 and POWER6 systems:
Implements a software Ethernet switch Supports up to 4094 networks (VLAN IDs) Copies packets between partitions (memory to memory) Consistent with VLAN IEEE 802.1Q standard The ports on this switch are configured via the HMC and are part of the partitions configuration

LPAR 1

LPAR 2

LPAR 3

LPAR 4

LPAR 5

VLAN 1

VLAN 2

VLAN 3 POWER Hypervisor (switch)

2008 IBM Corporation

IBM Training

Virtual Ethernet Adapters

Virtual Ethernet adapters are configured using the HMC or IVM, in the partitions profile or dynamically with DLPAR
Select the virtual slot number and PVID Configures the port for this slot on the Hypervisor switch Select additional VLANs Allows the adapter to communicate on multiple VLANS (up to 20 plus PVID) Requires additional configuration in the LPAR

Virtual Ethernet interfaces are configured in AIX just like physical adapter interfaces
2008 IBM Corporation

IBM Training

Virtual Ethernet Demonstration

Configure and test a virtual network between two AIX LPARs
POWER6 System VIOS LPAR1 LPAR2

192.168.1.1

192.168.1.2

en0

en0

ent0
PVID=1 ieee_virtual_eth=0

ent0
PVID=1 ieee_virtual_eth=0 POWER Hypervisor

VLAN 1

2008 IBM Corporation

IBM Training

Integrated Virtual Ethernet (IVE)

LPARs

Logical Switch

Logical Switch

Port Group 1 10 Gb or 1 Gb physical port 1 Gb physical port External Layer 2 or Layer 3 switch

Three IVE models: Dual-port Gigabit: 1 port group, 2 physical 1 Gb ports Dual-port 10 Gigabit: 2 port groups, 1 physical 10 Gb port per port group Quad-port Gigabit: 2 port groups, 2 physical 1 Gb ports per port group
2008 IBM Corporation

IBM Training

IVE Ports
Logical ports are associated with a specific physical port Port group: Set of 16 logical ports on 1 or 2 physical ports: Logical ports can be split evenly between the 2 physical ports in a port group, or unevenly 1 or 2 port groups per Host Ethernet Adapter (HEA), depending on model 1 or 2 physical ports per port group, depending on model, each with own Layer 2 switch LPAR Logical Port Logical Switch OS ent lhea Logical devices as they appear in AIX IVE

Port Group

HEA

Physical Port

External Switch
2008 IBM Corporation

IBM Training

Multi-Core Scaling (MCS)

The MCS value sets the maximum number of logical ports for a port group Set per port group (regardless of number of physical ports) Use MCS to specify number of ports you need to support LPARs Default value is 4, so only 4 logical ports per port group available Best IVE adapter performance when MCS value equal to # of processors Must power off/on managed system to change MCS

MCS = 1 Port Group

MCS = 4 Port Group

Switch

Switch Physical Switch

16 ports

Physical Switch

4 ports
2008 IBM Corporation

IBM Training

Managed System IVE Configuration (1 of 2)

1. Select the managed system, and choose: Hardware Information -> Adapters -> Host Ethernet
Physical port settings (actual) 2. Choose which IVE adapter

3. Choose physical port

4. Click Configure button to configure physical port

LPARs which are configured for logical ports using this physical port

2008 IBM Corporation

IBM Training

Managed System IVE Configuration (2 of 2)

4. button is used to configure physical port settings:

Enabled means HMC will attempt to negotiate flow control for physical port

MCS

Set to None or to a partition that will take entire physical port

A Virtual I/O Server partition must be listed as the promiscuous LPAR if it is using a port on the IVE as part of a Shared Ethernet adapter device
2008 IBM Corporation

IBM Training

LPAR IVE Configuration

Configure logical ports from the LPAR profile or with DLPAR

2008 IBM Corporation

IBM Training

IVE Demonstration
Configure an IVE logical port on the VIOS LPAR
1. 2. Check the configuration of the physical adapter Add a logical port for the VIOS LPAR
POWER6 System VIOS LPAR1 LPAR2

192.168.1.1

192.168.1.2

ent0 lhea0

en0 ent0
PVID=1 ieee_virtual_eth=0

en0 ent0
PVID=1 ieee_virtual_eth=0 POWER Hypervisor

Ethernet Switch

VLAN 1

P0

P1

HEA
2008 IBM Corporation

IBM Training

Shared Ethernet Adapter (SEA)

Shared Ethernet adapter bridges external networks to internal VLANs:
Forwards frames at OSI Layer 2 and is transparent to IP layer
Virtual I/O Server Partition Layer 2 Bridge (Shared Ethernet adapter)
Device Driver Physical Adapter External LANs Device Driver Virtual Adapter Device Driver Virtual Adapter 1-16 virtual adapters which connect to VLANs (and client partitions)
2008 IBM Corporation

IBM Training

SEA Example
Shared Ethernet adapter is used to extend VLAN 2 and 3 using the physical Ethernet in the Virtual I/O Server
Standalone servers Virtual I/O Server LPAR 1, 2, 3

Shared Ethernet VLAN 3 VLAN 2 VLAN 3 VLAN 2

Ethernet Switch

Managed system

2008 IBM Corporation

IBM Training

Configure Virtual Ethernet Adapter for SEA

Specify a PVID and optionally any VLAN IDs that the SEA will extend to an external network - PVIDs are stripped from
frames in the SEA, so use a unique VLAN for PVID when supporting access to VLANs on an external network

Virtual adapters that will be associated with an SEA on the VIO Server must have the Access external network checkbox selected

Virtual Ethernet adapters and Hypervisor switch ports are configured here on the HMC/IVM
2008 IBM Corporation

IBM Training

Configure SEA Device in VIO Server

Command to create Shared Ethernet Adapter:

$ mkvdev sea ent0 vadapter ent1 \ default ent1 defaultid 1

Creates device (ent2 in this case):

$ lsdev virtual name status ent1 ent2 Available Available

description Virtual I/O Ethernet Adapter (l-lan) Shared Ethernet Adapter

VIOS
switch

ent0 physical

ent1 virtual

PVID 1

Hypervisor

ent2 (SEA)

2008 IBM Corporation

IBM Training

SEA Demonstration
Configure an SEA adapter on the VIOS
1. 2. Create a virtual Ethernet adapter on the VIOS LPAR that can access external networks Create the SEA on the VIOS LPAR using the lhea port and the virtual adapter
POWER6 System VIOS SEA
192.168.1.1 192.168.1.2

LPAR1

LPAR2

ent0 lhea0 ent1

PVID=1 ieee_virtual_eth=0

en0 ent0
PVID=1 ieee_virtual_eth=0

en0 ent0
PVID=1 ieee_virtual_eth=0 POWER Hypervisor

Ethernet Switch
P0 P1

VLAN 1

HEA
2008 IBM Corporation

IBM Training

SEA with Link Aggregation

Create link aggregation Ethernet adapter:
$ mkvdev lnagg ent0 ent1

ent3 shows up as:

$ lsdev adapter | grep ent3 ent3 Available EtherChannel / IEEE 802.3ad Link Aggregation

Create Shared Ethernet adapter:

$ mkvdev sea ent3 vadapter ent2 \ default ent2 defaultid 1

Virtual I/O Server

switch
ent0 ent1 (Physical) ent3 (Aggregate) ent2 (Virtual) PVID 1 Hypervisor

ent4 (SEA)

2008 IBM Corporation

IBM Training

Virtual SCSI Overview

Virtual I/O Server
PHY PHY PHY PHY
VTD VTD VTD VTD VTD

Client

Client

Client

Physical Storage

Hypervisor

The red connections show two clients accessing the same physical storage (A) via two different server adapters (B) and virtual target devices (D) The blue connection shows multiple target devices (D) attached to a single server adapter (B)

PHY

Physical Adapter VSCSI Server Virtual Adapter

VSCSI Client Virtual Adapter

VTD

Virtual Target Device

2008 IBM Corporation

IBM Training

Virtual SCSI Configuration (1 of 3)

1) Define virtual SCSI server in VIO Server partition and client adapter in AIX or Linux partition

2) Check availability of virtual SCSI server adapters on VIO Server:

$ lsdev -virtual name status vasi0 Available vhost0 vsa0 Available Available description Virtual Asynchronous Services Interface (VASI) Virtual SCSI Server Adapter LPAR Virtual Serial Adapter
2008 IBM Corporation

IBM Training

Create vSCSI Adapters Demonstration

Create virtual SCSI adapters on the VIOS and AIX LPARs
POWER6 System VIOS LPAR1

hdisk7 hdisk6

hdisk5 hdisk0

cd0 vhost0 vscsi0

fcs0

fcs1

sas0

POWER Hypervisor

SAN

FC card

Internal storage

Optical device

2008 IBM Corporation

IBM Training

Virtual SCSI Configuration (2 of 3)

3) On the VIO Server, define storage resources

To create a volume group:

$ mkvg [ -f ][ -vg VolumeGroup ] PhysicalVolume ...

To create a logical volume:

$ mklv [ -lv NewLogicalVolume | -prefix Prefix ] VolumeGroup Size [PhysicalVolume ... ]

To create a storage pool:

$ mksp [-f] StoragePool PhysicalVolume ...

To create a backing device from available space in a storage pool:

$ mkbdsp [-sp StoragePool] Size [-bd BackingDevice] -vadapter ServerVirtualSCSIAdapter

2008 IBM Corporation

IBM Training

Virtual SCSI Configuration (3 of 3)

4) On the VIO Server, define virtual target devices
$ mkvdev -vdev TargetDevice -vadapter VirtualServerAdapter [ -dev DeviceName ]

For example:
$ mkvdev vdev hdisk3 vadapter vhost0 vtscsi0 Available $ mkvdev vdev lv10 vadapter vhost0 vtscsi1 Available $ mkvdev vdev cd0 vadapter vhost0 vtopt0 Available

Check the target devices with lsdev:

$ lsdev -virtual name status vtscsi0 Available vtscsi1 Available vtopt0 Available description Virtual Target Device - Disk Virtual Target Device - Logical Volume Virtual Target Device Optical Media

5) Boot the client or run cfgmgr to use new virtual devices

2008 IBM Corporation

IBM Training

View Configuration with lsmap

Use lsmap from the VIO Server to verify mapping of virtual targets:
$ lsmap -vadapter vhost0 SVSA vhost0 Physloc U9111.520.10F191F-V3-C6 Client Partition ID 0x00000003

--------------- ----------------------------------- -----------------

Client LPAR ID

Server slot ID
VTD LUN Backing device Physloc VTD LUN Backing device Physloc VTD LUN Backing device Physloc vtopt0 0x8300000000000000 cd0 vtscsi0 0x8100000000000000 hdisk3 U787A.001.DNZ00G0-P1-T10-L8-L0 vtscsi1 0x8200000000000000 lv10

Physical location code

LUN ID

2008 IBM Corporation

IBM Training

View Configuration with lshwres

Use lshwres from the HMC to see system-wide view of virtual I/O configuration (or view from HMC GUI)
hscroot@skylab-hmc:~> lshwres -r virtualio --rsubtype scsi -m skylab
lpar_name=VIOS,lpar_id=1,slot_num=7,state=1,is_required=1,ad apter_type=server remote_lpar_id=4,remote_lpar_name=node3,remote_slot_num=6, "backing_devices=drc_name=U787F.001.DPM0ZFL-P1-T10-L4L0/log_unit_num=0x8100000000000000/ device_name=hdisk1,drc_name=U787F.001.DPM0ZFL-P1-T10-L5L0/log_unit_num=0x820000000000000/ lpar_name=node3,lpar_id=4,slot_num=6,state=1,is_required=1,a dapter_type=client,remote_lpar_id=1,remote_lpar_name=VIOS,re mote_slot_num=7,backing_devices=none

2008 IBM Corporation

IBM Training

Virtual Target Device Demonstration

Configure virtual target devices for the vSCSI adapter with different physical backing devices
POWER6 System VIOS cl_lv clientVG
hdisk7 hdisk6 hdisk5 vtscsi1 vtopt0 vtscsi0

LPAR1 cd0 hdisk1 hdisk0 vscsi0

hdisk0

fcs0

fcs1

sas0

cd0

vhost0

POWER Hypervisor

SAN FC card Internal storage Optical device

2008 IBM Corporation

IBM Training

File-Backed Virtual Devices

File-back (FB) virtual device types:
File-backed disk devices
Files created in storage pools can be used as hdisk on client

File-backed optical media devices

Create a Virtual Media Repository which can be stocked with DVD-ROM/RAM media Clients can use images stored in repository as cd0 devices with media FB virtual device characteristics:
Read-only FB devices can be shared by multiple clients Bootable FB devices appear in SMS Reside in FB storage pools

Mount Directory = /var/vio/storagepools/<FBSP_Name> LV_NAME = <FBSP_Name>

Granularity as small as 1MB or as large as parent Logical Volume

FB virtual devices are new as of Virtual I/O Server V1.5

2008 IBM Corporation

IBM Training

Creating File-Backed Virtual Disks

Files on the virtual I/O Server can be used as backing storage:
1. 2. 3. 4. Create a volume group (mkvg) or storage pool (mksp -f) Create a FB disk storage pool (mksp -fb) inside volume group/storage pool Create a device in the pool (mkbdsp) and map to a vadapter The client associated with that vadapter sees new FB device as an hdisk
Volume Group/Storage Pool - contains hdisk(s) FB Disk Storage Pool (contains FB virtual disks)
Target dev Target dev Target dev

2008 IBM Corporation

IBM Training

Create FB Virtual Disks Example (1 of 2)

Create new volume group/logical volume storage pool:
$ mkvg -vg newvg hdisk1 OR mksp -f newvg hdisk1

New storage pool (newvg)

Create new FB storage pool in the logical volume storage pool:

$ mksp -fb fbpool -sp newvg -size 10g fbpool File system created successfully. 10444276 kilobytes total disk space. New File System size is 20971520

New FB storage pool (fbpool) that is 10 GB inside of newvg

Create new file device with a certain size, create the VTD, and map to vhost adapter: Create new 30 MB file called fb_disk1
$ mkbdsp -sp fbpool 30m -bd fb_disk1 -vadapter vhost3 Creating file "fb_disk1" in storage pool "fbpool". Assigning file "fb_disk1" as a backing device. vtscsi3 Available fb_disk1

Resulting VTD is named vtscsi3 and is mapped to vhost3

2008 IBM Corporation

IBM Training

Create FB Virtual Disks Example (2 of 2)

View mapping with new backing device:
$ lsmap -vadapter vhost3 SVSA ID Physloc Client Partition

--------------- ----------------------------- ----------------vhost3 VTD Status LUN Backing device Physloc U8203.E4A.10CD1F1-V1-C15 vtscsi3 Available 0x8100000000000000 /var/vio/storagepools/fbpool/fb_disk1 0x00000000

2008 IBM Corporation

IBM Training

Create FB Virtual Optical Device (1 of 2)

Create volume group/logical volume storage pool:
$ mkvg -vg medrep hdisk4 OR mksp -f medrep hdisk1 New storage pool (medrep)

Create 10 GB Virtual Media Repository in the LV pool:

$ mkrep -sp medrep -size 10G Virtual Media Repository Created Repository created within "VMLibrary_LV" logical volume

Create media (aixopt1) in repository from a file:

Media could be blank, loaded from cd# device, or a file
$ mkvopt -name aixopt1 -file dvd.product.iso -ro

2008 IBM Corporation

IBM Training

Create FB Virtual Optical Device (2 of 2)

View repository and its contents:

$ lsrep Size(mb) Free(mb) Parent Pool Free 10198 6532 medrep 59648 Name aixopt1
Create FB virtual optical device and map to vhost adapter:

Parent Size 69888

Parent

File Size Optical 3666 None

Access ro

$ mkvdev -fbo -vadapter vhost4 vtopt0 Available New VTD name

Load the image into the media device:
Use the unloadopt command to unload

$ loadopt -vtd vtopt0 -disk aixopt1

2008 IBM Corporation

IBM Training

Viewing FB Configuration from the HMC

HMC command line example:

hmc:~> lshwres -m hurston -r virtualio --rsubtype scsi lpar_name=VIOS,lpar_id=1,slot_num=16,state=1,is_required=0,adapte r_type=server,remote_lpar_id=any,remote_lpar_name=,remote_slot_nu m=any,"backing_devices=""0x8100000000000000//""""/var/vio/VMLibra ry/aixopt1""""""" . . .
2008 IBM Corporation

IBM Training

FB Device Command Examples (1 of 2)

List the repository and any contents:
$ lsrep

Size(mb) Free(mb) Parent Pool 10198 Name aixopt1 6532 medrep

Parent Size 69888 File Size Optical 3666 vtopt0

Parent Free 59648 Access ro

List the storage pools:

Notice both LVPOOL and FBPOOL types:
$ lssp Pool rootvg NewVG medrep fbpool Size(mb) 69888 69888 69888 10199 Free(mb) 44544 59648 59648 6072 Alloc Size(mb) 128 64 64 64 BDs Type 1 LVPOOL 0 LVPOOL 0 LVPOOL 2 FBPOOL

List out volume groups/storage pools (LVPOOL type only):

$ lsvg rootvg NewVG medrep
2008 IBM Corporation

IBM Training

FB Device Command Examples (2 of 2)

List LVPOOL details:
$ lssp -detail -sp NewVG Name PVID hdisk3 000cd1f195f987df Size(mb) 69888

List FBPOOL details:

$ lssp -bd -sp fbpool Name fb_disk1 fb_disk2 Size(mb) VTD 30 vtscsi3 4096 vtscsi4 SVSA vhost3 vhost3

Show all mounts including FB devices:

$ mount node mounted mounted over vfs date options -------- --------------- --------------- ------ ------------ --------------/dev/hd4 / jfs2 Apr 18 13:01 rw,log=/dev/hd8 /dev/hd2 /usr jfs2 Apr 18 13:01 rw,log=/dev/hd8 /dev/hd9var /var jfs2 Apr 18 13:01 rw,log=/dev/hd8 /dev/hd3 /tmp jfs2 Apr 18 13:01 rw,log=/dev/hd8 /dev/hd1 /home jfs2 Apr 18 13:01 rw,log=/dev/hd8 /proc /proc procfs Apr 18 13:01 rw /dev/hd10opt /opt jfs2 Apr 18 13:01 rw,log=/dev/hd8 /dev/fbpool /var/vio/storagepools/fbpool jfs2 Apr 28 12:04 rw,log=INLINE /dev/VMLibrary_LV /var/vio/VMLibrary jfs2 Apr 28 14:36 rw,log=INLINE

2008 IBM Corporation

IBM Training

File-Backed Virtual Devices Demonstration

Configure a file-backed virtual disk and file-backed virtual optical device
VIOS
fb_disk1

LPAR1 vtscsi2

fbpool1
(FB storage pool)

fb_disk2

rootvg medrep
cl_mksysb AIX53_iso AIX61_iso

stpool1
(LV storage pool)

(Virtual Media Repository)

vtopt1

hdisk2

hdisk1 hdisk0 vhost1

cd1 vscsi1

POWER Hypervisor

2008 IBM Corporation

IBM Training

Conclusion
Virtual I/O devices on POWER5 and POWER6 systems include virtual Ethernet, Integrated Virtual Ethernet (IVE), Shared Ethernet (SEA), and virtual SCSI Virtual Ethernet and IVE, are part of the base feature and do not require PowerVM (or Advanced Power Virtualization for POWER5)
Devices are defined in the HMC Configuration in AIX is same as for physical devices

Virtual SCSI and SEA devices extend the use of physical devices to multiple clients using the Virtual I/O Server and require the PowerVM feature
Create virtual devices with mkvdev on VIO Server command line to bridge virtual Ethernet adapters (mkvdev sea) and to associate virtual SCSI adapters with physical storage

File-backed virtual disks and optical devices provide even more flexibility in storage allocation
2008 IBM Corporation

IBM Training

Trademarks
The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.
Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*, AS/400, e business(logo), DBE, ESCO, eServer, FICON, IBM, IBM (logo), iSeries, MVS, OS/390, pSeries, RS/6000, S/30, VM/ESA, VSE/ESA, WebSphere, xSeries, z/OS, zSeries, z/VM, System i, System i5, System p, System p5, System x, System z, System z9, BladeCenter

The following are trademarks or registered trademarks of other companies.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce. * All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

39

2008 IBM Corporation