STORAGE ARCHITECTURE/ GETTING STARTED: SAN SCHOOL 101

Marc Farley President of Building Storage, Inc Author, Building Storage Networks, Inc.

Agenda
Lesson 1: Basics of SANs Lesson 2: The I/O path Lesson 3: Storage subsystems Lesson 4: RAID, volume management and virtualization Lesson 5: SAN network technology Lesson 6: File systems

Lesson #1

Basics of storage networking

Connecting

Concentrator Router

Dish

HBA or NIC Computer System

Network Switch/hub Bridge

VPN

Connecting
Networking or bus technology Cables + connectors System adapters + network device drivers Network devices such as hubs, switches, routers Virtual networking Flow control Network security

Storing

Host Software

Volume Manager Software

Mirroring Software

Storage Device Drivers

Storage Protocol

Storage Command and Transfer Protocol

(wiring, network transmission frame)

Storage Devices

Tape Drives

Disk Drives RAID Subsystem

Storing
Device (target) command and control
Drives, subsystems, device emulation

Block storage address space manipulation (partition management)

Mirroring RAID Striping Virtualization Concatentation

Filing
C:\directory\file
Database Object

User/Application View

User/Application View

Logical Block Mapping (Storage)

Filing
Namespace presents data to end users and applications as files and directories (folders) Manages use of storage address spaces Metadata for identifying data
file name owner dates

Connecting, storing and filing as a complete storage system

Filing

Wiring Connecting
HBA or NIC
Cable Cable

Storing

Network Switch/hub

Storing function in an HBA driver

Computer System

Disk Drive

NAS and SAN analysis

NAS is filing over a network SAN is storing over a network NAS and SAN are independent technologies They can be implemented independently They can co-exist in the same environment They can both operate and provide services to the same users/applications

Protocol analysis for NAS and SAN

NAS

Filing Storing Storing Wiring Connecting

SAN Network

Integrated SAN/NAS environment

NAS Client
NAS Head Server NAS + Server SAN Initiator NAS Head System

SAN SAN Target Storage

Filing Filing Wiring Connecting

Storing Storing
Wiring Connecting

Common wiring with NAS and SAN

NAS Client NAS Head Server NAS Head System

SAN Storage Target

Filing Filing Wiring Connecting

Storing Storing

Lesson #2

The I/O path

Host hardware path components

Memory

Processor

Memory Bus

System I/O Bus

Storage Adapter (HBA)

Host software path components

Application Operating Filing Cache Volume System System Manager Manager

MultiPathing

Device Driver

Network hardware path components

Cabling Fiber optic Copper

Switches, hubs, routers, bridges, gatways Port buffers, processors Backplane, bus, crossbar, mesh, memory

Network software path components

Access and Security

Fabric Services

Routing

Flow Control

Virtual Networking

Subsystem path components

Network Ports

Access and Security

Cache

Resource Manager

Internal Bus or Network

Device and media path components

Disk drives

Tape drives

Tape Media

Solid state devices

The end to end I/O path picture

Memory Cache Volume MultiFiling System Manager Manager Pathing Memory System I/O Bus Bus Device Storage Driver Adapter (HBA)

App

Processor Operating System

Cabling

Network Systems

Access and Security

Fabric Services

Routing

Flow Control

Virtual Networking

Subsystem Network Poirt

Access and Security

Cache

Resource Manager

Internal Bus or Network

Disk drives

Tape drives

Lesson #3

Storage subsystems

Generic storage subsystem model

Controller (logic+processors) Access control Network Ports Resource manager
Storage Resources

Cache Memory

Internal Bus or Network

Power

Redundancy for high availability

Multiple hot swappable power supplies Hot swappable cooling fans Data redundancy via RAID Multi-path support
Network ports to storage resources

Physical and virtual storage

Exported storage Exported storage

Physical storage device

Physical storage device

Exported storage

Exported storage

Exported storage

Exported storage

Physical storage device

Subsystem Controller Resource Manager (RAID, mirroring, etc.)

Physical storage device

Exported storage

Exported storage

Hot Spare Device

SCSI communications architectures determine SAN operations

SCSI communications are independent of connectivity SCSI initiators (HBAs) generate I/O activity They communicate with targets Targets have communications addresses Targets can have many storage resources Each resource is a single SCSI logical unit (LU) with a universal

unique ID (UUID) - sometimes referred to as a serial number An LU can be represented by multiple logical unit numbers (LUNs) Provisioning associates LUNs with LUs & subsystem ports

A storage resource is not a LUN, its an LU

Provisioning storage
LUN 0 LUN 1
SCSI LU UUID A

Port S1
LUN 1 LUN 2 LUN 2

Physical storage devices

SCSI LU UUID B

Port S2

Port S3

LUN 3 LUN 3

SCSI LU UUID C

Physical storage devices

Port S4

LUN 0

SCSI LU UUID D

Physical storage devices

Controller functions

Multipathing

LUN X

Path 1
SCSI LU UUID A

MP SW
LUN X

Path 2

Caching
Exported Volume Exported Volume

Controller Cache Manager

Exported Volume Exported Volume

Read Caches 1. Recently Used 2. Read Ahead

Write Caches 1. Write Through (to disk) 2. Write Back (from cache)

Tape subsystems
Tape Drive Tape Drive Tape Drive Tape Drive

Tape Subsystem Controller

Tape Slots

Robot

Subsystem management

Now Now with with SMIS SMIS

Management station browser-based network mgmt software

Ethernet/TCP/IP
Out-of-band management port

In-band management

Storage Subsystem

Exported Storage Resource

Data redundancy

Duplication

2n

Parity

n+1
Difference

-1
d(x) = f(x) f(x-1) f(x-1)

Duplication redundancy with mirroring

Host-based Within a subsystem

I/O Path

Mirroring Operator Terminate I/O & regenerate new I/Os Error recovery/notification

I/O PathA I/O PathB

Duplication redundancy with remote copy

Host Uni-directional (writes only)

Point-in-time snapshot

Subsystem Snapshot
Host

Lesson #4

RAID, volume management and virtualization

RAID = parity redundancy

Duplication

2n

Parity

n+1
Difference

-1
d(x) = f(x) f(x-1) f(x-1)

History of RAID Late 1980s R&D project at UC Berkeley

David Patterson Garth Gibson

(independent) Redundant array of inexpensive disks

Striping without redundancy was not defined (RAID 0)

Original goals were to reduce the cost and increase the capacity of large disk storage

Benefits of RAID

Capacity scaling

Combine multiple address spaces as a single virtual address

Performance through parallelism

Spread I/Os over multiple disk spindles

Reliability/availability with redundancy

Disk mirroring (striping to 2 disks) Parity RAID (striping to more than 2 disks)

Capacity scaling
Combined extents 1 - 12 Storage extent 1 Storage extent 2 Storage extent 3 Storage extent 4

Exported RAID disk volume (1 address)

Storage extent 5

Storage extent 6

Storage extent 7

Storage extent 8

RAID Controller (resource manager)

Storage Storage extent 9 extent10

Storage extent11

Storage extent12

Performance
RAID controller (microsecond performance)

Disk drive

Disk drive

Disk drive

Disk drive

Disk drive

Disk drive

Disk drives (Millisecond performance) from rotational latency and seek time

Parity redundancy
RAID arrays use XOR for calculating parity
Operand 1 False False True True Operand 2 False True False True XOR Result False True True False

XOR is the inverse of itself

Apply XOR in the table above from right to left Apply XOR to any two columns to get the third

Reduced mode operations

When a member is missing, data that is accessed must be reconstructed with xor An array that is reconstructing data is said to be operating in reduced mode System performance during reduced mode operations can be significantly reduced

XOR {M1&M2&M3&P}

Parity rebuild

The process of recreating data on a replacement member is called a parity rebuild Parity rebuilds are often scheduled for non-production hours because performance disruptions can be so severe

RAID Parity Rebuild

XOR {M1&M2&M3&P}

RAID 0+1, 10
RAID Controller Hybrid RAID: 0+1

DiskDisk drive drive 1

Disk Disk drive drive 2

Disk Disk drive drive 3

Disk Disk drive drive

Disk Disk drive drive 5

4 Mirrored pairs of striped members

Volume management and virtualization

Storing level functions Provide RAID-like functionality in host systems and SAN network systems Aggregation of storage resources for:
scalability availability cost / efficiency manageability

OS kernel

File system

Volume management

RAID & partition management Device driver layer between the kernel and storage I/O drivers

Volume Manager

HBA drivers HBAs

Server system

Volume managers can use all available connections and resources and can span multiple SANs as well as SCSI and SAN resources

Virtual Storage

Volume manager HBA drivers

SCSI disk resource

SCSI HBA SCSI Bus SAN cable

SAN disk resources SAN Switch

SAN HBA

SAN storage virtualization

RAID and partition management in SAN systems Two architectures:

In-band virtualization (synchronous) Out-of-band virtualization (asynchronous)

In-band virtualization
Exported virtual storage

SAN virtualization I/O Path system

System(s), switch or router

Disk subsystems

Out-of-band virtualization

Virtualization management system

Distributed volume management

Virtualization agents

Virtualization agents are managed from a central system in the SAN

Disk subsystems

Lesson #5

SAN networks

Fibre channel
The first major SAN networking technology Very low latency High reliability Fiber optic cables Copper cables Extended distance 1, 2 or 4 Gb transmission speeds Strongly typed

Fibre channel
A Fibre Channel fabric presents a consistent interface and set of services across all switches in a network Host and subsystems all 'see' the same resources

SAN Storage Target Subsystem

SAN Storage Target Subsystem

SAN Storage Target Subsystem

Fibre channel port definitions

FC ports are defined by their network role

N-ports: end node ports connecting to fabrics

L-ports: end node ports connecting to loops NL-ports: end node ports connecting to fabrics or loops F-ports: switch ports connecting to N ports FL-ports: switch ports connecting to N ports or NL ports in a loop E-ports: switch ports connecting to other switch ports G ports: generic switch ports that can be F, FL or E ports

Ethernet / TCP / IP SAN technologies

Leveraging the install base of Ethernet and TCP/IP networks iSCSI native SAN over IP FC/IP FC SAN extensions over IP

iSCSI
Native storage I/O over TCP/IP
New industry standard Locally over Gigabit Ethernet Remotely over ATM, SONET, 10Gb Ethernet

iSCSI TCP IP MAC PHY

iSCSI equipment

Storage NICs (HBAs)

SCSI drivers

Cables
Copper and fiber

Network systems
Switches/routers Firewalls

FC/IP
Extending FC SANs over TCP/IP networks FCIP gateways operate as virtual E-port connections FCIP creates a single fabric where all resources appear to be local

E-port

FCIP FCIP Gateway Gateway

One TCP/IP fabric

LAN, MAN or WAN

FCIP FCIP Gateway Gateway

E-port

SAN switching & fabrics

High-end SAN switches have latencies of 1 - 3 sec Transaction processing requires lowest latency
Most other applications do not

Transaction processing requires non-blocking switches

No internal delays preventing data transfers

Switches and directors

Switches
8 48 ports Redundant power supplies Single system supervisor

Directors
64+ ports HA redundancy Dual system supervisor Live SW upgrades

SAN topologies Star

Simplest single hop

Dual star Simple network

+ redundancy

Single hop Independent or integrated fabric(s)

SAN topologies N-wide star

Scalable Single hop Independent or integrated fabric(s)

Core - edge Scalable 1 3 hops integrated fabric

SAN topologies Ring

Scalable integrated fabric 1 to N2 hops

Ring + Star Scalable integrated fabric 1 to 3 hops

Lesson #6

File systems

File system functions

Name space Access control Metadata Locking Address space management

Filing Filing

Storing Storing

Think of the storage address space as a sequence of storage locations (a flat address space)
1 2 10 11 19 20 28 29 37 38 46 . 55 . 64 . 73 . 82 83

3
12 21 . . . . . . 84

4
13 . . . . . . . 85

6 7
16 . . . . . . . 88

8
17 26 35 44 53 62 71 80 89

9
18 27 36 45 54 63 72 81 90

14 15 . . . . . . . . . . . . . . 86 87

Superblocks are known addresses used to find Superblocks file system roots (and mount the file system)

SB

SB

File systems must have a known and Filing and Scaling dependable address space
The fine print in scalability - How does the filing function know about the new storing address space?

1 7 13 19 25 31 37

2 8 14 20 26 32 38

3 9 15 21 27 33 39

4 10 16 22 28 34 40

5 11 17 23 29 35 41

6 12 18 24 30 36 42

Filing Filing

Storing Storing Storing Storing

1 6 11 16 21

2 7 12 17 22

3 8 13 18 23

4 9 14 19 24

5 10 15 20 25