How To Connect Oracle Exalogic Elastic Cloud To IB Switch

An Oracle White Paper
October 2013
How to Connect Oracle Exalogic Elastic Cloud to

10 G Networks Using Oracle’s Ethernet
Switches
How to Connect Oracle Exalogic Elastic Cloud to 10 G Networks Using Oracle’s Ethernet Switches
Introduction ....................................................................................... 2
Exalogic Elastic Cloud X3-2 Full Rack Configuration ......................... 3
Multiple Rack Configuration of Exalogic Elastic Cloud X3-2 with
High Availability ................................................................................. 4
How to Configure the Network for Oracle Exalogic Elastic Cloud....... 6
Sun Network QDR InfiniBand Gateway Switch to Sun Network
10 GbE Switch 72p ........................................................................ 6
Sun Network 10 GbE Switch 72p to Client’s 10 G Network—L2
Uplinks .......................................................................................... 8
Sun Network 10 GbE Switch 72p to Client’s 10 G Network—L3
Uplinks ........................................................................................ 10
Conclusion ...................................................................................... 13
References .................................................................................. 14
Introduction
This paper provides instructions on how to connect different types of Oracle

Exalogic Elastic Cloud systems to an existing 10 G Ethernet network using
Oracle’s Ethernet switches. The paper also includes a description and sample
commands that show how to configure the network ports when connecting
Oracle Exalogic to a 10 G network.
Oracle Exalogic is an integrated hardware and software system designed to

provide a complete platform for a wide range of application types and widely
varied workloads. Oracle Exalogic is intended for large-scale, performance-
sensitive, mission-critical application deployments. Oracle Exalogic is designed
to fully leverage an internal InfiniBand fabric that connects all of the
processing, storage, memory, and external network interfaces within an Oracle
Exalogic machine to form a single, large computing device. Each Oracle
Exalogic machine is connected to a customer's data center networks via
10 GbE (traffic) and GbE (management) interfaces.
Oracle’s Exalogic Elastic Cloud X3-2—the hardware component of Oracle

Exalogic—can be deployed in either a single-rack or multiple-rack
configuration. This paper describes how to connect each of these two
configurations to an existing 10 G network. For each configuration, there is
also a detailed parts list of the required networking components so readers can
easily purchase all the necessary equipment.
Oracle Exalogic systems can also be connected to gigabit Ethernet networks,

but this topic will be covered in a separate white paper and is not discussed in
this paper.
2
Exalogic Elastic Cloud X3-2 Full Rack Configuration

Exalogic Elastic Cloud X3-2 is an engineered system with compute nodes and storage nodes. Its
Exabus feature provides architecture for InfiniBand and Ethernet connectivity. The 10 GbE network
connectivity is achieved through Oracle’s Sun Network QDR InfiniBand Gateway Switch. Exalogic
Elastic Cloud X3-2 systems come with four of these switches. Each gateway switch has two QSFP
(four 10 G) ports. These QSFP ports can be connected to 10 G ports on the client’s network using
QSFP splitter cables.
The proposed networking configuration uses Oracle’s Sun Network 10 GbE Switch 72p as an
aggregate switch for connecting the four gateway switches to the client’s 10 G network. Sun Network
10 GbE Switch 72p has 16 QSFP (four 10G) ports and eight 10G SFP+ ports. It is an ultra-dense,
non-blocking, low-latency switch with an industry-standard L2 and L3 feature set. It can be managed
through Oracle Integrated Lights Out Manager (Oracle ILOM). The data sheet and product related
information for Sun Network 10 GbE Switch 72p is available here.
Sun Network 10 GbE Switch 72p is a top-of-rack switch used as a server access switch with the 4x2
QSFP connections from the gateway switches terminating at the Sun Network 10 GbE Switch 72p.
The QSFP ports on the gateway switches come with the short range (SR) transceivers. The 8x10 G
SFP+ ports on Sun Network 10 GbE Switch 72p are reserved for uplink to the aggregate/core
network.
Figure 1. 10 GbE networking configuration for Exalogic Elastic Cloud X3-2, single full rack
3
Table 1 shows all the components required for this proposed networking configuration.
TABLE 1: COMPONENTS REQUIRED FOR CONNECTING EXALOGIC ELASTIC CLOUD X3-2 FULL RACK TO 10 GBE NETWORK
QUANTITY PART NUMBER DESCRIPTION
1 X2074A-R/X2074A-F Sun Network 10 GbE Switch 72p
Connections for Sun Network 10 GbE Switch 72p to Sun Network QDR InfiniBand Gateway Switch
8 X2124A QSFP transceivers
8 Molex 106283-1010 MPO-MPO high-bandwidth optical cable, 10 m
Molex 106283-1050 MPO-MPO high-bandwidth optical cable, 50 m
10 GbE – SFP+ to Uplinks
8 Customer-supplied SFP+ transceivers, cables for uplinks
Multiple Rack Configuration of Exalogic Elastic Cloud X3-2 with

High Availability
When deploying four Exalogic Elastic Cloud X3-2 racks together, the proposed networking
configuration utilizes two Sun Network 10 GbE Switch 72p switches for high availability (HA). Each
Exalogic Elastic Cloud X3-2 rack has four switches (Sun Network QDR InfiniBand Gateway Switch),
each with 2x QSFP ports. The Sun Network 10 GbE Switch 72p switch has 16 (four 10 G) QSFP
ports that are connected to the gateway switches across all four racks with one QSFP port from each
gateway switch going to a QSFP port on each Sun Network 10 GbE Switch 72p for high availability.
The Sun Network 10 GbE Switch 72p switches have 8x10 G SFP+ ports dedicated for uplink.
4
Figure 2: 10 GbE high-availability networking configuration for Exalogic Elastic Cloud X3-2, multiple full racks
Table 2 shows all the components required for this proposed networking configuration.
TABLE 2: COMPONENTS REQUIRED FOR CONNECTING FOUR EXALOGIC ELASTIC CLOUD X3-2 FULL RACKS TO 10 GBE NETWORK
QUANTITY PART NUMBER DESCRIPTION
2 x2074A-F/x2074A-R Sun Network 10 GbE Switch 72p
Connections for Sun Network 10 GbE Switch 72p
10 GbE – SFP+ Multimode Fiber (10 G Base-SR) to Gateway Switches
16 X2124A QSFP transceivers
16 Molex 106283-1010 MPO-MPO high-bandwidth optical cable, 10 m
10 GbE – SFP+ to Uplinks
16 Customer-supplied SFP+ transceivers, cables
5
How to Configure the Network for Oracle Exalogic Elastic Cloud

This section provides an example showing how to configure the network connections when
connecting an Oracle Exalogic system to the client’s 10 G network using Oracle’s Sun Network
10GbE Switch 72p.
In the Oracle Exalogic Elastic Cloud system configuration, the Ethernet interfaces on the servers are
instantiated as VNICs (active and passive), associated with corresponding HCA ports (active and
passive). It is one VNIC per compute node per virtual local area network (VLAN) and the VNIC has
to be associated with a 10 G port on the Sun Network QDR InfiniBand Gateway Switch. Each VNIC
has its own MAC address and optional VLAN ID. The port on the Sun Network 10 GbE Switch 72p
switch connected to an external Ethernet port on the gateway switch can be configured as a trunk port
to allow traffic from multiple VLANs.
There are two steps in the network configuration process. The first step is to configure the ports on
the InfiniBand gateway (Sun Network 10 GbE Switch 72p). The second step is to configure the uplink
ports from Sun Network 10 GbE Switch 72p so that the ports are connected to the client’s 10G
network. These ports can be configured using L2 or L3 uplink connectivity.
Each of these aspects is discussed in more detail below.
Sun Network QDR InfiniBand Gateway Switch to Sun Network 10 GbE Switch 72p
Each Sun Network QDR InfiniBand Gateway Switch in the Oracle Exalogic Elastic Cloud system has
two QSFP ports. In a full-rack Oracle Exalogic Elastic Cloud configuration, there are four switches
(Sun Network QDR InfiniBand Gateway Switch), so a total of eight QSFP ports are connected to the
10 G Ethernet via the Sun Network 10 GbE Switch 72p switch. With the QSFP ports on the Sun
Network QDR InfiniBand Gateway Switch connected to the Sun Network 10 GbE Switch 72p switch,
you can configure VLANs, port-channels, or both.
Consider the scenario in Figure 3, which shows an Oracle Exalogic Elastic Cloud system connected to
Sun Network 10 GbE Switch 72p. VLANs 2 and 3 are configured on the Sun Network QDR
InfiniBand Gateway Switch. The QSFP ports on each Sun Network QDR InfiniBand Gateway Switch
are connected to the first four QSFP ports (ports 1–16) on the Sun Network 10 GbE Switch 72p
switches.
6
Figure 3: 10 GbE HA networking configuration for Exalogic Elastic Cloud X3-2, single full rack—L2 uplinks
Given below is the L2 configuration for Sun Network 10 GbE Switch 72p.
Enable/disable default features and then configure the VLANs:
SEFOS# configure terminal

SEFOS(config)# set gvrp disable
SEFOS(config)# set gmrp disable
SEFOS(config)# set port-channel enable
SEFOS(config)# interface vlan 1
SEFOS(config-if)# shutdown
SEFOS(config-if)# no ip address
SEFOS(config-if)# end
Step 1: Create VLANs 2 and 3 on Sun Network 10 GbE Switch 72p.

SEFOS(config)# vlan 2
SEFOS(config-vlan)# ports add extreme-ethernet 0/1-16
SEFOS(config-vlan)# vlan active
SEFOS(config-vlan)# exit
SEFOS(config-vlan)# end
SEFOS# write startup-config
7
Step 2 (optional): Configure the ports connecting to the gateway switches (Sun Network QDR
InfiniBand Gateway Switch) as trunk ports if more VLANs need to be created in the future and if the
ports should carry them automatically (VLAN still needs to be created explicitly).

SEFOS(config)# interface range extreme-ethernet 0/1-4
SEFOS(config-if)# description "To Rack1-IBGW1"
SEFOS(config-if)# switchport mode trunk
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS# write startup-config
Sun Network 10 GbE Switch 72p to Client’s 10 G Network—L2 Uplinks

The 8x10 G SFP+ ports on Sun Network 10 GbE Switch 72p are reserved for uplink connection to
the client’s 10 GbE network.
To configure the uplink ports to the core switches, follow the guidelines below. In a high-availability
setup with two core switches, connect two 10 G ports from Sun Network 10 GbE Switch 72p to each
of them as shown in Figure 3. The uplink ports are aggregated and configured in trunk mode to allow
traffic for VLANs 2 and 3 to be carried.
Note: If the client’s core switches provide only 1 G connectivity, then the setting on the uplink posrts
should be “speed 1000.” At 1 G speed, the ports operate in forced-1000-FDX mode; hence,
auto-negotiation MUST be disabled on the client’s core switch ports connecting to the uplink ports on
Sun Network 10 GbE Switch 72p.
Step 3: Configure port aggregation for uplink ports. Port-channel 10 aggregates ports 65–66 to core
switch1. Port-channel 11 aggregates ports 67–68 to core switch2.

SEFOS(config)# interface port-channel 10
8
SEFOS(config)# exit
SEFOS(config)# end
Step 4: Add ports to uplink port-channels.

SEFOS(config-if)# channel-group 10 mode active
SEFOS(config-if)# channel-group 11 mode active
Step 5: Configure uplink port-channels to allow all VLANs.

SEFOS(config)# exit
SEFOS(config)# end
Note: In the topology shown in Figure 3, if the core switches are running vPC, then port-channels 10
and 11 should be configured for LACP mode and the individual ports of port-channels 10 and 11
should connect to both switches:
Ports 65, 67 to core switch1

Ports 66, 68 to core switch2
For detecting and avoiding loops in the network, configure the MSTP protocol. However, if the core
switch is configured for PVRST (Cisco proprietary protocol), you can configure PVRST on Sun
Network 10 GbE Switch 72p as shown below. Check the spanning tree for each VLAN using the
command show spanning-tree.
Note: While using PVRST, the ports can be either access ports or trunk ports. By default, a trunk port
will become a member of all VLANs configured on the switch. VLAN pruning capability is not
currently supported.
9
Step 6: Configure L2- PVRST on Sun Network 10 GbE Switch 72p such that the switch is not the root
bridge for any VLANs configured.

SEFOS(config)# spanning-tree mode pvrst
Spanning Tree enabled protocol is MSTP, now MSTP is being shutdown
PVRST is started.
PVRST Module status is changed
SEFOS(config)# spanning-tree vlan 2 brg-priority 61440
SEFOS(config)# spanning-tree vlan 3 brg-priority 61440
SEFOS(config)# end
Step 7: Save the configurations.
SEFOS# copy running-config startup-config
Sun Network 10 GbE Switch 72p to Client’s 10 G Network—L3 Uplinks

Consider the topology shown in Figure 4, where the uplink ports from Sun Network 10 GbE Switch
72p connecting to the client’s core switch are configured as L3 uplinks.
Uplink ports 65–68 are aggregated into port-channel 1 and are members of VLAN 20. VLAN 20 is
assigned an IP address that is in the same subnet as the ports on the core switch connecting to the
uplink ports on Sun Network 10 GbE Switch 72p. Ports 1–16 on Sun Network 10 GbE Switch 72p
are connected to the gateway switches (Sun Network QDR InfiniBand Gateway Switch) and are
members of VLAN 10.
VLAN 10 is assigned an IP address, which would be the gateway IP address for the servers in VLAN
10. At the least, static routing should be set up on Sun Network 10 GbE Switch 72p to reach the
default external gateway. Additionally, if needed, dynamic routing protocols such as RIP and OSPF (in
accordance with the routing protocol defined on the core) could be configured.
The inter switch link (ISL) between the two switches is established over port 64 and VLAN 99 is
configured as ISL VLAN.
10
Figure 4: 10 GbE networking configuration with VRRP for Exalogic Elastic Cloud X3-2, single full rack—L3 uplinks
Enable/disable default features and then configure the VLANs:

SEFOS(config)# set gvrp disable
SEFOS(config)# set gmrp disable
SEFOS(config)# set lldp enable
SEFOS(config)# ip routing
SEFOS(config)# int vlan 1
SEFOS(config-if)# no ip address
SEFOS(config-if)# shut
SEFOS(config)# exit
Step 1: Configure untagged VLANs between the two Sun Network 10 GbE Switch 72p switches as
‘access VLAN’.

SEFOS(config-vlan)# ports add extreme-ethernet 0/64 name ISL
SEFOS(config)# interface extreme-ethernet 0/64
SEFOS(config-if)# switchport access vlan 99
11
Step 2: Configure VLAN 10 as tagged on the Sun Network 10 GbE Switch 72p switch on the ports
connecting to the gateway switches (Sun Network QDR InfiniBand Gateway Switch).

SEFOS(config-vlan)# ports add extreme-ethernet 0/64
SEFOS(config-vlan)# end
Step 3: Assign L3 addresses to ISL VLAN and VLAN 10.

SEFOS(config-if)# ip address 99.99.99.101 255.255.255.0
Step 4: Configure VRRP.

SEFOS(config)# router vrrp
SEFOS(config-router)# int vlan 99
SEFOS(config-router)# vrrp 1 ipv4 99.99.99.1
SEFOS(config-router)# vrrp 1 accept-mode enable
SEFOS(config-router)# vrrp 1 priority 110
SEFOS(config-router)# exit
SEFOS(config-router)# int vlan 10
SEFOS(config-router)# vrrp 1 ipv4 192.168.10.1
SEFOS(config-router)# vrrp 1 accept-mode enable
SEFOS(config-router)# vrrp 1 priority 110
SEFOS(config-router)# exit
Note: About priority, it will be configured only on the primary switch; other switches will have default
priority 100.
Note: The IP addresses for VLANs 10 and 99 will be x.x.x.101 on one switch and x.x.x.102 on the
other.
On the server side, the static route to reach interface VLAN 10 via VRRP address 192.168.10.1 should
be configured.
12
Step 5: Create the port-channel for the uplink ports and add members to the uplink port-channel.

SEFOS(config-if)# switchport access vlan 20
SEFOS(config)# exit
SEFOS(config)# interface extreme-ethernet 0/65-68
SEFOS(config-if)# channel-group mode 1 active
SEFOS(config)# end
Step 6: Assign IP addresses to VLAN 20.

Step 7: Set the default route to the external gateway reachable through the uplink port-channel 1.

SEFOS(config)#ip route 0.0.0.0 0.0.0.0 192.168.20.1 1
SEFOS(config)# exit
Step 8: Save the configurations.
SEFOS# copy running-config startup-config
Conclusion
There are many variables when connecting and configuring an Oracle Exalogic system to a data center
network. This document is an attempt to simplify that process and reduce risk for Oracle customers by
provided detailed instructions for how to use Oracle networking products with Oracle Exalogic.
By following the guidelines outlined in this paper, readers can properly setup and configure the
network environment to connect an Oracle Exalogic system to their existing 10 G network.
13
References
For more information, visit the web resources listed in Table 3.
TABLE 3. WEB RESOURCES FOR FURTHER INFORMATION
WEB RESOURCE DESCRIPTION WEB RESOURCE URL
Oracle Exalogic http://docs.oracle.com/cd/E18476_01/index.htm
Oracle's Sun Network 10 GbE Switch 72p www.oracle.com/us/products/networking/ethernet/top-of-rack-

switches/overview/
Sun Network QDR InfiniBand Gateway Switch http://my.oracle.com/site/pd/sss/products/networking/qdr-ib-gateway/index.htm
Sun Ethernet Fabric Operating System CLI Base http://docs.oracle.com/cd/E19934-01/html/E26513/

Reference Manual
14
How to Connect Oracle Exalogic Elastic Cloud Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
to 10 G Networks Using Oracle’s Ethernet
Switches This document is provided for information purposes only, and the contents hereof are subject to change without notice. This
October 2013, Version 1.0 document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in
Author: Savi Venkatachalapathy law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any
liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This
document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our
Oracle Corporation
prior written permission.
World Headquarters
500 Oracle Parkway
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Redwood Shores, CA 94065
U.S.A.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and
Worldwide Inquiries: are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are
Phone: +1.650.506.7000 trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0113
Fax: +1.650.506.7200
oracle.com

How To Connect Oracle Exalogic Elastic Cloud To IB Switch

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

How To Connect Oracle Exalogic Elastic Cloud To IB Switch

Încărcat de

Drepturi de autor:

Formate disponibile

An Oracle White Paper

How to Connect Oracle Exalogic Elastic Cloud to

This paper provides instructions on how to connect different types of Oracle

Oracle Exalogic is an integrated hardware and software system designed to

Oracle’s Exalogic Elastic Cloud X3-2—the hardware component of Oracle

Oracle Exalogic systems can also be connected to gigabit Ethernet networks,

Exalogic Elastic Cloud X3-2 Full Rack Configuration

QUANTITY PART NUMBER DESCRIPTION

1 X2074A-R/X2074A-F Sun Network 10 GbE Switch 72p

8 X2124A QSFP transceivers

8 Molex 106283-1010 MPO-MPO high-bandwidth optical cable, 10 m

Molex 106283-1050 MPO-MPO high-bandwidth optical cable, 50 m

Molex 106283-1100 MPO-MPO high-bandwidth optical cable, 100 m

10 GbE – SFP+ to Uplinks

8 Customer-supplied SFP+ transceivers, cables for uplinks

Multiple Rack Configuration of Exalogic Elastic Cloud X3-2 with

QUANTITY PART NUMBER DESCRIPTION

2 x2074A-F/x2074A-R Sun Network 10 GbE Switch 72p

Connections for Sun Network 10 GbE Switch 72p

10 GbE – SFP+ Multimode Fiber (10 G Base-SR) to Gateway Switches

16 X2124A QSFP transceivers

16 Molex 106283-1010 MPO-MPO high-bandwidth optical cable, 10 m

Molex 106283-1050 MPO-MPO high-bandwidth optical cable, 50 m

Molex 106283-1100 MPO-MPO high-bandwidth optical cable, 100 m

10 GbE – SFP+ to Uplinks

16 Customer-supplied SFP+ transceivers, cables

How to Configure the Network for Oracle Exalogic Elastic Cloud

SEFOS# configure terminal

Step 1: Create VLANs 2 and 3 on Sun Network 10 GbE Switch 72p.

SEFOS# configure terminal

SEFOS# configure terminal

Sun Network 10 GbE Switch 72p to Client’s 10 G Network—L2 Uplinks

SEFOS# configure terminal

Step 4: Add ports to uplink port-channels.

SEFOS# configure terminal

Step 5: Configure uplink port-channels to allow all VLANs.

SEFOS# configure terminal

Ports 65, 67 to core switch1

SEFOS# configure terminal

Step 7: Save the configurations.

SEFOS# copy running-config startup-config

Sun Network 10 GbE Switch 72p to Client’s 10 G Network—L3 Uplinks

Enable/disable default features and then configure the VLANs:

SEFOS# configure terminal

SEFOS# configure terminal

SEFOS# configure terminal

Step 3: Assign L3 addresses to ISL VLAN and VLAN 10.

SEFOS# configure terminal

Step 4: Configure VRRP.

SEFOS# configure terminal

SEFOS# configure terminal

Step 6: Assign IP addresses to VLAN 20.

SEFOS# configure terminal

SEFOS# configure terminal

Step 8: Save the configurations.

SEFOS# copy running-config startup-config

WEB RESOURCE DESCRIPTION WEB RESOURCE URL

Oracle Exalogic http://docs.oracle.com/cd/E18476_01/index.htm

Oracle's Sun Network 10 GbE Switch 72p www.oracle.com/us/products/networking/ethernet/top-of-rack-

Sun Network QDR InfiniBand Gateway Switch http://my.oracle.com/site/pd/sss/products/networking/qdr-ib-gateway/index.htm

Sun Ethernet Fabric Operating System CLI Base http://docs.oracle.com/cd/E19934-01/html/E26513/

S-ar putea să vă placă și