Data Center Facilities Consideration in Designing and Building Networks

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 1
Data Center Facilities

Consideration in Designing
and Building Networks
BRKDCT-2867
Session_ID
© 2006, Cisco Systems, Inc. All rights reserved. 1

Presentation_ID.scr
Agenda
DC Standard
DC Challenges
Cooling Issues and Solutions
Cabling Issues and Solutions
Power Issues and Solutions
DC Physical Considerations
Modular Access (example)
Consolidated I\O Architecture with Virtualization
Summary
Q&A
Session_ID
Data Center Standard
Session_ID

Presentation_ID.scr
DC Standard
ANSI/TIA-942
Telecommunications Infrastructure
Standard for Data Centers
TIA/EIA TIA/EIA TIA/EIA TIA/EIA

568 569 606 607
Copper & Fiber Pathways Administration Grounding
Cabling & Spaces & Bonding
ASHRAE IEEE 1100

Uptime ITE Grounding
Cooling/HVAC
Institute
TIA: Telecommunications Industry Association http://www.tiaonline.org/

Uptime Institite: http://uptimeinstitute.org/
Government work on server and DC Energy Efficiency:
http://www.energystar.gov/index.cfm?c=prod_development.server_efficiency
Session_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Presentation_ID Cisco Public 5
TIA-942 Logical Layout
Offices, Operations Access Access

Providers Entrance Room
Center, Support (Carrier Equip & Providers
Rooms Demarcation)
Horizontal Cabling
Backbone Cabling
Computer
Main Dist Area
Telecom Room Backbone Cabling (Routers/Backbone
Room
(Office & Operations LAN/SAN Switches,
Center LAN Switches) PBX, M13 Muxes) Backbone Cabling
Horiz Dist Area

(LAN/SAN/KVM
Switches)
Horizontal Cabling Horiz Dist Area Horiz Dist Area Horiz Dist Area
(LAN/SAN/KVM (LAN/SAN/KVM (LAN/SAN/KVM
Switches) Switches) Switches)
Zone Dist Area
Horizontal Cabling Horizontal Cabling Horizontal Cabling Horizontal Cabling
Equipment Dist Equipment Dist Equipment Dist Equipment Dist

Area Area Area Area
(Rack / Cabinet) (Rack / Cabinet) (Rack / Cabinet) (Rack / Cabinet)
Session_ID

Presentation_ID.scr
Example Data Center Layout
Main Distribution Area
Horizontal Distribution Area
Equipment Distribution Area
Session_ID
Zone Distribution Area (ZDA)
LEGEND:
EDA
(server cabinet)
Horizontal Cabling
(in hot aisles)
Patch Cord
(to server)
ZDA
(Zone Outlet or
Consolidation Point)
Session_ID

Presentation_ID.scr
Data Center Tiering
Excerpt from TIA-942 Standard
Tier II Tier III

Tier I Tier IV
Redundant Concurrently
Basic Fault Tolerant
Components Maintainable
Site Availability 99.671% 99.749% 99.982% 99.995%
Downtown(Hours/Year) 28.8 22.0 1.6 0.4
Operations Center Not Required Not Required Required Required
Redundant Access
Not Required Not Required Required Required
Provider Services
Redundant Backbone
No No Yes Yes
Pathways
Redundant Horizontal
No No No Optional
Cabling
UPS Redundancy N N+1 N+1 2N
Gaseous Suppression Clean Agents Clean Agents
No No
System FM200/Intergen FM200/Intergen
Session_ID
Data Center
Challenges
Session_ID

Presentation_ID.scr
Problem Statement
Systems Entering Data Centers Have Changed;
Most Legacy Server Environments Lack Sufficient
Infrastructure to Gracefully Handle Them
Blade servers Storage devices
80-84 severs in cabinet Large footprints
30kW of Power 1500, 2000, 2500+ lbs.
1U systems 700, 900, 1100+ kgs.
Greater port densities
Greater heat output
More weight
Session_ID
Data Center Environmental Challenges

Typical Shortcomings
Cooling
Structured cabling
Power
Structural loading
The nature of Data Center

infrastructure makes it
challenging to find solutions
that don’t spawn other
problems
Session_ID

Presentation_ID.scr
Data Center Inefficiencies
Uptime is the ultimate goal, as a result……

Data Centers are over-planned
To provide 24X7 availability
Allow for future growth
90% of corporate Data Centers have more cooling capacity
than required (Uptime Institute)
Inefficient equipment deployment
Server performance, one application per server
72% of cooling bypasses the computing equipment entirely
(Uptime Institute)
Inherent power inefficiencies
Session_ID
Cooling Issues and

Solutions
Session_ID

Presentation_ID.scr
Today’s products 2000 – 2010 Product Heat Density Trend Chart

are hotter than
yesterday’s
Tomorrow’s
products will be
hotter than today’s
Data Center
Managers prefer to
tightly install
equipment to fully
utilize cabinet space
Session_ID

Ambient temperatures
50, 100, 150, 200 watts/sq. ft.
500, 1000, 1600, 2150 watts/ sq. m.
Hot spots
Short-cycling of air handlers
Arrhenius Rate Law
For every 18 °F (10 °C) increase in temperature there is a
50% decrease in reliability of electronics
Activation Energy
Rate Constant
EA
-
Kelvin Temperature
K = AeRT
Frequency Factor
or Pre-Exponential The Gas Constant
Factor Mathematical
Session_ID
Quantity, e

Presentation_ID.scr
Design Solutions
Spread out heat sources
Increase aisle spaces
Don’t cluster high-density rows
Use deep plenum below floor and above ceiling

Orient air handlers perpendicular to rows
Session_ID
Catalyst Switches in a Cabinet
Catalyst 6K Switches in a
Cabinet
Utilize Ducting
Improve Air Flow
Characteristics
CFD of 2 – Catalyst 6509 CFD of 2 – Catalyst 6509

Switches with ducting Switches without ducting
Session_ID

Presentation_ID.scr
Airflow
Cold Aisle / Hot Aisle
Air Circulation Components Cold Aisle/Hot Aisle Concept
Session_ID Cable Placement Additional Cooling Capacity


Design Solutions
Hot / cold aisles Ducted air return
Chimney design
Session_ID

Presentation_ID.scr
Cooling Issues and Solutions -
Architectures
Central Air Handling Unit Computer Room Air Handler In-Row Air Handler
(CAHU) (CRAH) (IRAH)
Session_ID
Cooling Issues and Solutions - A Hybrid

Approach to Cooling
Most Data Centers will have a mix of heat densities and
therefore cooling solutions
IT Refreshes happen every 2-4 years resulting in a mix of distributed
IT assets
IT management of blades and storage deployments is often in clusters
Concentrated high density loads
Older server assets may be well served by room cooling units
Leverage existing cooling assets to maintain room conditions.
Row-
oriented Rack-
d
oriented
Room-
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. oriented
Cisco Public 22

Presentation_ID.scr
Cooling Issues and Solutions –
Alternative cooling architectures w/ power considerations
Method Application Density

Traditional room- Low density 1-5kW per rack
oriented raised floor Very flexible
cooling
In-row Medium density 3-15kW per rack
General use
In-row with hot aisle Very high density 10-25kW per rack
containment Targeted zones
Assured redundancy
Rack-coupled Very high density specific 20-45kW per rack

racks
Mix of very high and low
density
Session_ID

Design Solutions
Deploy redundant infrastructure

Supplemental cooling solutions
Overhead cooling units
Water-cooled cabinets
Auto-adjusting tile dampers
Cool Door Technology
In row cooling
Session_ID

Presentation_ID.scr
Operational Solutions
Distribute hottest systems

Limit clustering of like systems
Install air dams
Streamline cabling
Maintain static pressure
Employ temperature monitoring tools
Install air conservation system on floor
Session_ID
Cooling Issues and Solutions – In Row

Cooling (iRAH)
Elimination of mixing enables a
predictable cooling pattern
IT Racks
Capture heat at the source and
Neutralize
Capable of cooling high densities > Cooling
30 kW per rack
Close Coupled to Heat Load
Thermal Containment Options
Dynamic fan control matches heat
removal to heat generation
Redundancy – Reduce Power
Virtualization – Ramp-up and Down
to meet thermal demand
Reduced deployment cycle and
Cost through use of modular
scaleable components
Build out in Zones.
Cool only what you need to
Increased data center efficiency
Front View IRAH – In row air handler
Session_ID

Presentation_ID.scr
Cooling Issues and Solutions - Hot Aisle
Containment Systems (HACS)
Hot Aisle Ceiling
Tiles/Cable Trough
Seals in hot air, prevents
mixing with room air
High Density Zones
Supports InRow products
Hot air scavenging system
Ducted Return / Free Supply
Optimize InRow Cooling
Increase Efficiency
Improve Predictability
Use at any density Chamber Doors
InfraStruXure InRow RC
Access to hot aisle, locks for
In-Row Air Conditioner Cools hot security
chamber air
InfraStruXure High Density

Session_ID
Data Center Cooling Solutions Summary

1. Conduct a cooling checkup/survey.
2. Route data cabling in the hot aisles
and power cable in the cold aisles.
3. Control air path leaks and manage
cabling system pathways.
4. Remove obstructions below raised
floor and seal cutouts.
5. Separate blade server cabinets.
6. Implement ASHRAE TC9.9 hot
aisle/cold aisle design.
7. Place CRAC units at the ends of the
hot aisles.
8. Manage floor vents.
9. Install air flow assisting devices as
needed.
10. In extreme cases, consider self-
contained cooling units.
Session_ID

Presentation_ID.scr
Cabling Issues and
Solutions
Session_ID
Insufficient ports
Users “borrow” other connections
Piecemeal fixes
Chaotic cabling
Restricts air flow
Hinders troubleshooting
Creates unplanned dependencies
Prone to accidental downtime
Mess begets mess
Session_ID

Presentation_ID.scr
Design Solutions
Right-size port counts

Set capacity to meet 95% of deployments
Prewire cabinet locations
Plan cable management

Choose graceful high-density solutions
Strategically deploy wire management
Employ a distributed physical design
Session_ID

Direct-Connect Design
One main networking row
Cabling routed directly to
server cabinet locations
Excellent for the logical
elements of the network
Session_ID

Presentation_ID.scr
Bad for the physical element
of a network
Scales poorly
Prone to cable overlap
Session_ID

Distributed Design
Network substations
Cabling to all server cabinets
Subset of cables to the main
network row
Session_ID

Presentation_ID.scr
Design Solutions
Distributed Design
Good for the physical
element of a network
Scales well
No cable overlap
Cable runs are shorter
and better organized
Easier to manage
Less expensive
Less restrictive for air flow
Session_ID
Cabling Issues and Solutions: Pathways
Cable Sizes Growing

Cat 5e 0.157”
Cat 6 0.25”
Cat 6a 0.315”
Multiple Diverse Routes
Under the Access Floor
Above the Racks \ Cabinets
Mixture of Under & Over the Access Floor
Session_ID

Presentation_ID.scr
Cabling Issues and Solutions:
Pathways and Spaces Under Floor
Benefits
Pedestals create infrastructure pathways
Utilization of real estate
Cabling is hidden
Concerns
Could restrict cold airflow
Creating segregated pathways
Accessibility to cables
Session_ID
Cabling Issues and Solutions:

Pathways and Spaces Overhead
Benefits
Alleviates congestion beneath access floor
Creation of segregated pathways
Minimizes restrictions to cold air floor
Concerns
Requires adequate space above the racks
Infrastructure provisions to support the pathways
Cabling may be exposed
Session_ID

Presentation_ID.scr
Before After
Session_ID

- Zone Cabling in the DC
Flexibility to Cable Full Rack Equipment Easily

Mainframes
SAN Equipment
Ability to make MACs Quickly and Easily
Reduced Network Downtime When Changes are Required
Session_ID

Presentation_ID.scr
- Cabling a Cisco Modular Switch
Which way do you cable switch?
Session_ID

Neatly route cabling

Don’t use overly long
patch cords
Use wire management as
designed
Session_ID

Presentation_ID.scr
Evolution of Ethernet Physical Media –
Impact on Facilities
Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s
10Mb 100Mb 1Gb 10Gb
UTP Cat 3 UTP Cat 5 UTP Cat 5 X2

SFP Fiber SFP+ Cu
SFP+ Fiber
Cat 6/7 ??
Power Transceiver
Technology Cable Distance (each side) Latency (link)
SFP+ CU
Twinax 10m ~0.1W ~0.1μs
Copper
SFP+ USR MM OM2 10m

1W ~0
ultra short reach MM OM3 100m
SFP+ SR MM 62.5μm 82m

1W ~0
short reach MM 50μm 300m
Cat6 55m ~8W 2.5μs

10GBASE-T Cat6a/7 100m ~8W 2.5μs
Cat6a/7 30m ~4W 1.5μs
Session_ID
Power Issues and

Solutions
Session_ID

Presentation_ID.scr
Growing demand for circuits
From: (2) 120V 20 amp circuits -Two poles
To: (2) 208V 30 amp circuits - Four poles
Growing demand for capacity

Today’s electrical loads exceed yesterday’s designs
Eliminates redundancy (N+1)
Exceeds total building capacity
Session_ID

Design Solutions
Prewire all cabinet locations
Set capacity to meet 95% of deployments
Provide redundant power

Feed from multiple PDUs
Keep all component loads below 50%
Provide capacity to accommodate growth

Employ a distributed physical design
Session_ID

Presentation_ID.scr
Electrical conduits routed
directly to all cabinet
locations
Restrict airflow
Complicate troubleshooting
Create unintentional physical
dependencies
Overlapping conduits
In large Data Centers
conduit lengths can be
excessive
Session_ID

Distributed Design
Fewer overlapping
conduits
Improved airflow
Reduced costs to modify
shorter conduits
Reduced risk of multiple
outages from a physical
accident
Session_ID

Presentation_ID.scr
Electrical/Network Substation
Remote Power Panels
Installed back-to-back
Each fed from a different PDU
Network Patching Fields
Cabinets Floor Tiles Ports per Tile

1 15 24 Fiber, 18 Copper
2 24-30 36 Fiber, 36 Copper
5 50 48 Fiber, 48 Copper
Session_ID

Use hosts with redundant power supplies
Limit systems with odd-numbered power cords
Consider amp-reading power strips
Session_ID

Presentation_ID.scr
Data Center Power Considerations Summary
1. See Cooling top 10 Steps!
2. Standardize on rack SOE
3. Implement scalable UPS systems
4. Increase Voltage
5. Targeted higher UPS loading
6. Investigate power efficiency
7. Load balance
8. Limit branch circuit proliferation
9. Monitor power
10. Manage and target power based
on monitoring benchmark
Session_ID
DC Physical
Considerations
Session_ID

Presentation_ID.scr
Data Center Topology
Network Equipment and Zones
Core
Layer
Aggregation
Layer
Access
Layer
DC
Zone
Pod
Pod
Network Rack
HOT AISLE
Server Rack
COLD AISLE
Storage Rack Pod
Session_ID
Pod Concept
Network Zones and Pods
HOT AISLE
Pod
COLD AISLE
Pod
Sizing
▪Zone: Typically mapped to agg pair size
▪Pod: Typically mapped to access switch pair
▪ Size: determined by distance and density
▪ Cabling distance from server racks to network racks
▪ 100m Copper
▪ 200-500m Fiber
▪ Cabling density: # of servesr per rack and I/Os per server
▪Rack
▪ Server: 6-30 Servers per rack – limited by power
▪ Network: Depends on access model: Modular, ToR or Blade
▪ Storage: special Cabinets
Session_ID

Presentation_ID.scr
Network Equipment Distribution
End of Row End of Row
End of Row
▪Traditionally used
▪Copper from server to access switches Patch panel Patch panel
▪Poses challenges on highly dense server farms
Patch panel Patch panel
▫ Distance from farthest rack to access point X-connect X-connect
server server
▫ Row length may not lend itself well to
switch port density
Network Network
Access Point Access Point
Common Characteristics A-B C-D
▪Typically used for modular access server server
▪Cabling is done at DC build-out
▪Model evolving from EoR to MoR
▪Lower cabling distances (lower cost) Fiber
▪Allows denser access (better flexibility)
▪6-12 multi-RU servers per Rack Copper
▪4-6 kW per server rack, 10Kw-20Kw per network End of Row
rack
▪Subnets and VLANs: one or many per switch.
Subnets tend to be medium and large
End of Row (half row) server

Patch panel
X-connect
Patch panel
server
X-connect
▪Use is starting to increase given EoR
challenges
▪Copper from servers to access switches
▪Fiber may be used to aggregate ToR
Network Network
▪It addresses aggregation requirements for Access Point Access Point
ToR access environments A-B C-D
server server
Session_ID

Top of Rack
ToR Patch panel Patch panel

▪Used in conjunction with dense access Top of Rack Top of Rack
racks(1U servers) server
server
X-connect X-connect
▪Typically one access switch per rack
▪Some customers are considering two +
cluster
▪Typically: Network
Aggregation
Network
Aggregation
▪ ~10-15 server per rack (enterprises) Point Point
▪ ~15-30 server per rack (SP) server
A-B A-B
server
▪ Use of either side of rack is gaining traction
▪ Cabling:
▪Within rack: Copper for server to access To network core
switch
▪Outside rack (uplink):
▪Copper (GE): needs a MoR model for fiber
aggregation
▪Fiber (GE or 10GE):is more flexible and also
requires aggregation model (MoR) Top of Rack Top of Rack
▪Subnets and VLANS: Top of Rack
X-connect X-connect
Top of Rack
▪ one or many subnets per access switch server server
▪ Subnets tent to be small
Network Network
Access Point Access Point
A-B C-D
server server
Session_ID

Presentation_ID.scr
Blade Chassis
End of Row (Switch to Switch)

▪Scales well for blade server racks (~3 blade
chassis per rack) Patch panel Patch panel
▪Most current uplinks are copper but the NG
switches will offer fiber Patch panel Patch panel
sw1 sw2 X-connect X-connect sw1 sw2
Blade Chassis Blade Chassis
End of Row (Pass-through)
▪Scales well for pass-through blade racks sw1 sw2 sw1 sw2
▪Copper from servers to access switches Blade Chassis Network Network Blade Chassis
Aggregation Aggregation
Point Point
sw1 sw2 A–B–C-D A–B-C-D sw1 sw2
ToR Patch panel

Patch panel
▪Have not seen it used in conjunction with Top of Rack
blade switches Patch panel Patch panel
▪May be a viable option on pass-through Pass-through X-connect X-connect Pass-through
environments is the access port count is right Blade Chassis Blade Chassis
Pass-through Pass-through
Blade Chassis Network Network Blade Chassis
Aggregation Aggregation
Point Point
Pass-through A–B–C-D A–B-C-D Pass-through
Session_ID
Width of Cabinets
24” vs 32” Cabinets?
For Each Scenario in this example

96in x 42in footprint
Total of 16.8KW of Power
Total of 48 Servers
Air Flow delivered is the same
32in 45RU Cabinet Advantages
Reduced Static pressure due to proper cable
management with more than 4-5 2RU servers per
cabinet
Vertical patch panels minimizes patch cord lengths,
number of sizes, & increases usable RU spacing
Power cables and Network cables have good
separation
Additional room for cabling thus reducing accidental
downtime
Footprint only increases with the number
of servers in a 24” cabinet.
17 – 2RU Servers
34 – 1RU Servers
Session_ID

Presentation_ID.scr
Differences between Cabinets and Racks
• Aesthetics
• Security
• Additional Usable Cable Management
Area & Flexibility
• Vertical Patch Panels
• Footprint
• Additional Cooling Options
• Cool Door
• Convert back and forth from Server to
Switch cabinet
Session_ID
Risks to Consider in Capacity Planning

Legacy Server High-Density Server
Power per Server 2-3 kW per rack > 20 kW per rack
Power per Floor Space 30-40 W/ft² 700-800 W/ft²
Cooling Needs—chilled airflow 200-300 cfm 3,000 cfm
Source: Gartner 2006
20,000 ft²
Legacy DC designed to accommodate

800kW 2-3kW per Rack
+33%
Introducing 1/3 high-density infrastructure into

a legacy facility has cost, power, weight, and
100-200 Racks cooling implications *Peripheral DC costs considered
Session_ID

Presentation_ID.scr
Modular Access / End
Row Example
Session_ID
End of Row Example (Modular Access)

The Challenge
Data Center
Core
~4000 Servers
Aggregation
Server LAN A&B Connectivity Aggregation
Module
Layer
Servers 1Gb Connectivity
Uplinks 10Gb Connectivity
Utilize Catalyst 6509 Switches 2
2 2 2 2
2 2 2
Core, Agg and Access Design 48 Switches
SAN Cabinets
4000 Servers
Mainframe\Midrange Cabinets
450 Watts per Server Access Layer
5.5 kW per Switch Cabinet

4 kW per SAN\Midrange Cabinet 10 GbE
GE
“POD” Concept Design
Session_ID

Presentation_ID.scr
Data Center Sizing
Enterprise Modular Access: 6509
Enterprise
Servers 1500 < x < 4000 Core:
8000 6509: 16 10GE ÷ 4 ports per LC = 4 LCs
Ports*
2 LCs for cross connectivity
Switch Types 6509
Uplinks: based on DC outbound requirements
Core 2: 6509
Aggregation:
Switches LCs: 6 4x10GE
6509: 96x10GE ÷ 4 ports per LC = 24 LCs
4: 6509s
2 LCs for cross connectivity
Aggregation LCs: 26 4x10GE
Switches Outbound Capacity: 80GE 26 LCs ÷ 8 Slots = 3.25 ~ 4 chassis

Oversubscription: 96:16 4 chassis x 4 uplinks = 16 10GE ports
Access 24: 6509 Access: 8000 ÷ 336 = 23.8 ≈ 24 chassis
Switches Oversubscription: 8.4:1 4 10GE uplinks per chassis = 96 10GE
Maximum Capacity Oversubscription: 336:40 ~ 8.4:1
Access: 336 x 24 = 8064 ports ~ 4032 Servers

4 10GE uplinks per chassis = 8 10GE
Oversubscription: 336:40 ~ 8.4:1
Session_ID

The Layout and Sample Solution
12 Server “PODs” Servers: 4032
Consists of the following: 6509 Switches: 30
4 Switch Cabinets for LAN & SAN Server\Switch Cabinets: 399
32 Server Cabinets Midrange\SAN Cabinets Allotted For: 124
12 Servers per Server Cabinet
Core 1 Core 2
Agg1 Agg2 Agg3 Agg4
Acc1 Acc2 Acc11 Acc12 Acc13 Acc14 Acc23 Acc24
6 Pair 6 Pair
Switches Switches
336 Servers 336 Servers 336 Servers 336 Servers

Session_ID

Presentation_ID.scr
The Data Center Layout Total White Space:
14,400 sqFt
Session_ID

The “POD” with EDA Breakout
Equipment Distribution
Area (EDA)
Single “POD”
Acc1 Acc2
336 Servers
Session_ID

Presentation_ID.scr
EDA Application Photos
Session_ID

The “POD” with HDA Breakout
Horizontal Distribution
Area (HDA)
Single “POD”
Acc1 Acc2
336 Servers
Session_ID

Presentation_ID.scr
The “POD” with HDA Breakout
Horizontal Distribution
Area (HDA)
Single “POD”
Acc1 Acc2
336 Servers
Session_ID

HDA Application Photos
Session_ID

Presentation_ID.scr
Main Distribution Area (MDA)
Core 1 Core 2
Additional Equipment:
Core Routing\Firewalls
Agg1 Agg2 Agg3 Agg4

LAN Appliances
SAN Directors
Session_ID

Main Distribution Area (MDA)
Additional Equipment
Core 1 Core 2
Core Routing\Firewalls
LAN Appliances
SAN Directors
Session_ID

Presentation_ID.scr
CFD Analysis
Designed in a Hot –
Cold Architecture
12 - 20 Ton CRAC Units

Outside
12 - 30 To CRAC Units
Inside
Utilizing Ceiling plenum

for return air
All Perforated Tiles at

25% Open
Peak Temp was 114°
Session_ID
Consolidated I/O
Architecture with
Virtualization
Session_ID

Presentation_ID.scr
Increased Efficiency, Simpler Operations
Mgmt
Network
Front-End
Network
Backup
Network
Unified
Fabric
Storage Back-End
Network Network
Unified Fabric and I/O
Session_ID
Fibre Channel over Ethernet (FCoE)
A method for a direct mapping of FC frames over

Ethernet
Seamlessly connects to FC networks Æ Extends FC across the datacenter
over the Ethernet
FCoE appears as FC to the host and the
network
Preserves current FC infrastructure 10Gb Etherne
t
and management Pipe
FC
FC frame is unchanged
Can operate over standard switches
(with jumbo frames)
Priority Flow Control guarantees no-drops
Mimics FC credit-buffer system, avoids TCP
Does not require expensive off-loads
Session_ID

Presentation_ID.scr
VM-Optimized Services
Enables convergence of SAN A SAN B
LAN LAN
multiple traffic types
Virtual Machines LAN
Virtual Machines SAN
Hypervisor Mgmt LAN
Virtual Infrastructure Services LAN
Scales VM LAN performance

Increase I/O bandwidth
Increase VM density VMotion
Accelerates Virtual
Infrastructure Services
Live VM migrations via VMotion and 10 GbE
DRS features 10 GbE DCE
Enable additional services Fibre Channel
10 GbE
FCoE/DCE
Session_ID
Virtualized Server Environment – Unified Fabric

ToR Deployment
40 Servers per Cabinet Pair VMs per Server: 10
2 Nexus 5020 per Cabinet Pair Servers: 400
10 Cabinet Pairs Nexus 7000 Switches: 2
Nexus 5000 Switches: 10
MDS Switches: 2
Agg1 Agg2
Acc1 Acc2 Acc1 Acc2
10 CP
40 Servers 40 Servers
Session_ID

Presentation_ID.scr
A Single “POD” Using Virtualized Model
24 Cabinets Total
• 10 Server Cabinet Pairs (20 Total)
• 4 Switch Cabinets
Over Head Fiber Optic Cabling Only

400 Servers with 10 Virtualized Each (4000
Total Virtual Servers)
Session_ID

The Cabinet Pair Breakout
Cross Patching
between Cabinets
20 Servers per Cabinet
X 2 Cabinets (40 Total)
40 Connections per
Switch
Each Switch Uplinks
• 4 – LAN A \ 4 – LAN B
• 4 – SAN A \ 4 – SAN B
Session_ID

Presentation_ID.scr
Unified Fabric - Consolidated I/O Extends
Benefits of ToR
Consolidate LAN I/O Consolidate SAN I/O
• Consolidate multiple LAN adapters • No need for SAN-specific adapters
into two CNAs • Reduce SAN switches and cables
• Consolidate multiple cables into two • Unified I/O switches connect to
paths existing SAN infrastructure
Unconsolidated I/O Consolidated I/O
2x SAN
Consolidated I/O Benefits 2x Consolidated
2x LAN
• CapEx Savings up to 30%
• Cable reduction of 50% or
more for ToR and EoR
designs
• Potential power savings
Session_ID
Virtualization Impact on Critical Facilities.
Power Needed per Server has increased

Could be up to 16KW for this cabinet
Need for Supplemental Cooling required
Weight of Equipment on Raised Floor
All Copper Cabling is Contained with the Two Cabinets
Cable reduction
Fiber Optic Cabling for the Uplinks
The need for quality Fiber Cabling has increased
Session_ID

Presentation_ID.scr
Summary
Session_ID

Typical Shortcomings
Cooling
Structured cabling
Power
Structural loading
The nature of Data Center

infrastructure makes it
challenging to find solutions
that don’t spawn other
problems
Session_ID

Presentation_ID.scr
Universal Solutions
Physically Distribute
Equipment
Power & Cooling
Right-size infrastructure
(modularity)
Virtualize
Network Architecture and
Facilities Dependencies
Use pools of servers and

storage, controlled by the
network, to provide Data
Center resources.
Session_ID
Universal Solutions
Service Oriented Data Center (SODC)
More efficient use of computing resources
Shared computing among multiple machines
Built-in redundancy
Individual servers are expendable
Devices can be managed on a by-cabinet basis
Session_ID

Presentation_ID.scr
Q and A
Session_ID
Recommended Reading
Topics
Site selection
Data Center sizing and layout
Physical design
Large-scale server moves
Remote monitoring
Change management
Includes downloadable
design template
Available Onsite at the Cisco Company Store

Session_ID

Presentation_ID.scr
Recommended Reading
Continue your Networkers at Cisco
Live learning experience with further
reading from Cisco Press
Check the Recommended Reading
flyer for suggested books
Cisco Press
Data Center Fundamentals
Session_ID
Available Onsite at the Cisco Company Store
Additional Resources
URLs
6500 Cabinet Information
http://wwwin.cisco.com/dss/isbu/6500/enviro/index.shtml
Panduit
http://www.panduit.com/default.asp
TIA – Telecommunications Industry Association
http://www.tiaonline.org/
ASHRAE – American Society of Heating, Refrigerating and Air-Conditioning Engineers
http://www.ashrae.org/
Uptime Institute
http://uptimeinstitute.org/
Government work on server and DC Energy Efficiency:
http://www.energystar.gov/index.cfm?c=prod_development.server_efficiency
Session_ID

Presentation_ID.scr
Complete Your Online
Session Evaluation
Win fabulous prizes; give us

your feedback
Receive ten Passport Points
for each session evaluation
you complete
Go to the Internet stations
located throughout the
Convention Center to complete
your session evaluation
Winners will be announced
daily at the Internet stations
Session_ID
Session_ID

Presentation_ID.scr

Data Center Facilities Consideration in Designing and Building Networks

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Data Center Facilities Consideration in Designing and Building Networks

Încărcat de

Drepturi de autor:

Formate disponibile

Session_ID

Data Center Facilities

© 2006, Cisco Systems, Inc. All rights reserved. 1

Data Center Standard

© 2006, Cisco Systems, Inc. All rights reserved. 2

TIA/EIA TIA/EIA TIA/EIA TIA/EIA

ASHRAE IEEE 1100

TIA: Telecommunications Industry Association http://www.tiaonline.org/

TIA-942 Logical Layout

Offices, Operations Access Access

Horiz Dist Area

Equipment Dist Equipment Dist Equipment Dist Equipment Dist

© 2006, Cisco Systems, Inc. All rights reserved. 3

Main Distribution Area

Horizontal Distribution Area

Equipment Distribution Area

Zone Distribution Area (ZDA)

© 2006, Cisco Systems, Inc. All rights reserved. 4

Tier II Tier III

© 2006, Cisco Systems, Inc. All rights reserved. 5

Data Center Environmental Challenges

The nature of Data Center

© 2006, Cisco Systems, Inc. All rights reserved. 6

 Uptime is the ultimate goal, as a result……

Cooling Issues and

© 2006, Cisco Systems, Inc. All rights reserved. 7

 Today’s products 2000 – 2010 Product Heat Density Trend Chart

Cooling Issues and Solutions

© 2006, Cisco Systems, Inc. All rights reserved. 8

 Use deep plenum below floor and above ceiling

Catalyst Switches in a Cabinet

CFD of 2 – Catalyst 6509 CFD of 2 – Catalyst 6509

© 2006, Cisco Systems, Inc. All rights reserved. 9

Air Circulation Components Cold Aisle/Hot Aisle Concept

Session_ID Cable Placement Additional Cooling Capacity

Cooling Issues and Solutions

© 2006, Cisco Systems, Inc. All rights reserved. 10

Cooling Issues and Solutions - A Hybrid

© 2006, Cisco Systems, Inc. All rights reserved. 11

Method Application Density

Rack-coupled Very high density specific 20-45kW per rack

Cooling Issues and Solutions

 Deploy redundant infrastructure

© 2006, Cisco Systems, Inc. All rights reserved. 12

 Distribute hottest systems

Cooling Issues and Solutions – In Row

© 2006, Cisco Systems, Inc. All rights reserved. 13

InfraStruXure High Density

Data Center Cooling Solutions Summary

© 2006, Cisco Systems, Inc. All rights reserved. 14

Cabling Issues and Solutions

© 2006, Cisco Systems, Inc. All rights reserved. 15

 Right-size port counts

 Plan cable management

 Employ a distributed physical design

Cabling Issues and Solutions

© 2006, Cisco Systems, Inc. All rights reserved. 16

Cabling Issues and Solutions

© 2006, Cisco Systems, Inc. All rights reserved. 17

Cabling Issues and Solutions: Pathways

 Cable Sizes Growing

© 2006, Cisco Systems, Inc. All rights reserved. 18

Uptime is the ultimate goal, as a result……

Today’s products 2000 – 2010 Product Heat Density Trend Chart

Use deep plenum below floor and above ceiling

Deploy redundant infrastructure

Distribute hottest systems

Right-size port counts

Plan cable management

Employ a distributed physical design

Cable Sizes Growing

Flexibility to Cable Full Rack Equipment Easily

Ability to make MACs Quickly and Easily

Reduced Network Downtime When Changes are Required

Neatly route cabling

Growing demand for capacity

Provide redundant power

Provide capacity to accommodate growth

Network Patching Fields

For Each Scenario in this example

Core, Agg and Access Design 48 Switches

5.5 kW per Switch Cabinet