Interconnection Networks

Dr. Rajender Nath

Dr. Rajender Nath

Interconnection Networks
How to connect individual devices together Device Device Device into a community of communicating devices? Device SW Interface SW Interface SW Interface SW Interface Device (definition): HW Interface HW Interface HW Interface HW Interface Component within a computer Single computer System of computers Types of elements: Interconnection Network Interconnection Network end nodes (device + interface) links interconnection network
Link Link Link Link End Node End Node End Node End Node

Dr. Rajender Nath

Interconnection Network Domains

Interconnection networks can be grouped into four major networking domains, depending on the number and proximity of devices to be interconnected: OCNs, SANs, LANs, and WANs On-Chip Networks (OCNs), Network-on-Chip (NoC)
Interconnect microarchitecture functional units, register caches, compute tiles, processor and IP cores Chip or multichip modules Tens (in future, possibly 100s) of devices interconnected Maximum interconnect distance on the order of centimeters Examples (custom designed) files,

Element Interconnect Bus (Cell Broadband Engine processor chip) 2,400 Gbps (3.2 Ghz processor clock), 12 elements on the chip

Examples (proprietary designs)

CoreConnect (IBM), AMBA (ARM), Smart Interconnect (Sonic)

Dr. Rajender Nath

On Chip Networks
On-chip network

Dr. Rajender Nath

Interconnection Network Domains

System/storage area networks (SANs)
Multiprocessor and multicomputer systems
Interprocessor and processor-memory interconnections

Server and data center environments

Storage and I/O components

Hundreds to thousands of devices interconnected

IBM Blue Gene/L supercomputer (64K nodes, each with 2 processors)

Maximum interconnect distance typically on the order of tens of meters, but some with as high as a few hundred meters
InfiniBand: 120 Gbps over a distance of 300 m

Examples (standards and proprietary)

InfiniBand, Myrinet, Quadrics, Advanced Switching Interconnect

Dr. Rajender Nath

Interconnection Network (BlueGene)

Dr. Rajender Nath

Interconnection Network Domains

Local area networks (LANs)
Interconnect autonomous computer systems Machine room or throughout a building or campus Hundreds of devices interconnected (1,000s with bridging) Maximum interconnect distance on the order of few kilometers, but some with distance spans of a few tens of kilometers Hundred devices (thousands with bridging) Example (most popular): Ethernet, with 10 Gbps over 40Km

Wide area networks (WANs)

Interconnect systems distributed across the globe Internetworking support is required Many millions of devices interconnected Maximum interconnect distance of many thousands of kilometers Example: ATM Dr. Rajender Nath

Basic Network Structure and Functions

Media and Form Factor
Metal layers InfiniBand connectors Ethernet

Fiber Optics

Media type
Printed circuit boards

Cat5E twisted pair Coaxial cables

OCNs 0.01 1

SANs 10

Myrinet connectors

LANs 100

WANs >1,000

Distance (meters) Dr. Rajender Nath

Types of Interconnection Networks

Two kinds of Topologies
Direct/Static/point-to-point
Ratio of switch nodes to processor nodes is 1:1 Every switch node is connected to
1 processor node At least 1 other switch node

Indirect/Dynamic
Ratio of switch nodes to processor nodes is greater than 1:1 Some switches simply connect other switches
Dr. Rajender Nath

Static Interconnection Networks

Linear Array Ring Chordal Ring Barrel Shifter Tree and Star Fat Tree Mesh Torus Hyper Cubes Cube Connected Cycles K-ary n-Cube cycles

Dr. Rajender Nath

Dynamic Interconnection Network

Buses
Single Multiple Hierarchical

Cross-bar Multistage
Omega Butterfly Benes Baseline Generalized cube
Dr. Rajender Nath

Evaluating Interconnection Networks

Network Size (N): No. of nodes in the network Node Degree (d): No. of edges incident on a nodein-degree, out-degree Network Diameter (D): The distance between the farthest two nodes in the network. Bisection Width (B): The minimum number of wires you must cut to divide the network into two equal parts. No. of Links (l): Links/channels
Dr. Rajender Nath

Evaluating Interconnection Networks

Cost: The number of links or switches (whichever is asymptotically higher) is a meaningful measure of the cost. However, a number of other factors, such as the ability to layout the network, the length of wires, etc., also factor in to the cost.

Dr. Rajender Nath

Completely-connection Network
Each node has a direct communication link to every other node in the network. Ideal in the sense that a node can send a message to another node in a single step. Static counterpart of crossbar switching networks Nonblocking

Dr. Rajender Nath

Network Topologies: Completely Connected and Star Connected Networks

N=8 d=N-1 D=1

# of Links=N(N-1)/2 =>cost

Symmetric B=(N/2)2

Dr. Rajender Nath

Star-Connected Network

One Node acts as the central Node. Every other Node has a communication link connecting it to this central Node. Similar to bus-based network. The central Node is the bottleneck. Two-level tree

Dr. Rajender Nath

Star Connected Networks

N=9 d=N-1, d=1 D=2

# of Links=N-1 =>cost

Asymmetric B=N/2

Dr. Rajender Nath

Linear Array
N=5 d=2, d=1 D=N-1
# of Links=N-1 =>cost

Asymmetric B=1

Dr. Rajender Nath

Ring
N=8 d=2 D=N/2
# of Links=N =>cost

Symmetric B=2

Dr. Rajender Nath

Chordal Ring
N=8 d=3 D=3
# of Links=12 =>cost

Symmetric B=4-6

Dr. Rajender Nath

Chordal Ring
N=8 d=4 D=2
# of Links=16 =>cost

Symmetric B=6

Dr. Rajender Nath

Barrel Shifter
Number of nodes N = 2n Start with a ring Add extra edges from each node to those nodes having power of 2 distance i & j are connected if |j-I| = 2r, r = 0, 1, 2, , n-1

Dr. Rajender Nath

Barrel Shifter
N=8 d=2n-1 D=n/2 Symmetric

Dr. Rajender Nath

Tree Network

Dr. Rajender Nath

Binary Tree
N= d = 1,2,3 Height of the tree = h =log2N D=2(h-1) B=1 Asymmetric

Dr. Rajender Nath

Fat Tree

Dr. Rajender Nath

2D Mesh Network
N=nk = 9 k-dimensional mesh=2 d=2, 3, 4(2k) #of Links=kN-kn D=k(n-1) Asymmetric B=n
Dr. Rajender Nath

2D Torus Network
N=nk = 9 d=4(2k) #of Links=2N D=2n/2 Symmetric B=2n

Dr. Rajender Nath

Thank You