Sunteți pe pagina 1din 84

Size Matters: MTU Considerations

and Implications in the Data Center


Eric Spielman, Network Consulting Engineer
BRKDCN-1100

Agenda

Overview of MTU within the Data Center

Sizing across multiple protocols

MTU interaction with Data Center technologies

Ins & Outs of MTU related configuration

Summary

Session Goal

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Overview of MTU within the


Data Center

The Data Center Evolution

VXLAN, OTV

FabricPath
vPC / vPC+

STP
Fabric
Extension

Unified Fabric
FCoE, iSCSI

STP Tiered Design

Full Link Utilization

Host & Network Overlays

Separate SAN Infrastructure

Converged Fabrics

Increased Scalability

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Role of MTU

What is the maximum size of a


frame that a network device will
accept?

Bidirectional attribute

Hop-by-hop basis

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Jumbo Frames

Any payload size greater than


1500 bytes

Does not include Ethernet headers


Enables 6x as much payload data in
one transmit

14 byte
Eth Hdr
20 byte
IP Hdr
20 byte
L4 Hdr

14 byte
Eth Hdr
20 byte
IP Hdr
20 byte
L4 Hdr

Data Payload
1460 bytes
4 byte
Eth FCS

Data Payload
9176 bytes

Nexus & UCS support payloads of


up to 9216 bytes

MTU 1500
4 byte
Eth FCS

MTU 9216
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Why the Draw to Jumbo Frames?

Larger MTU is needed for protocols and technologies like FCoE, OTV,
and recommended for apps such as vMotion

Jumbo frames reduce the number of copies and context switches that
occur in the host due to packet overhead reduction

A 9000 byte frame results in up to 4 context switches and 4 copies to message


in, process, and exit the host
A typical 1500 byte frame would require 6x as many to receive the same
amount of data

Improved forwarding and buffer utilization from network devices

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Finding the Right Size


PMTUD for IPv4

MTU 9000
R1
7K-1
Host

MTU 9000

MTU 1500

R2

1. Host sends 9000 byte packet to discover


max MTU
2. R2-R3 link is only 1500 MTU & needs to
fragment
3. DF-bit prevents & ICMP unreachable is
sent back to Host with link MTU of 1500

7K-2

Server

Transmit with new MTU value

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

10

PMTUD
In real life...

ISP

7K-1

7K-2
(config-if)# no ip unreachables

Host

Server

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

11

Effects of MTU Mismatches

Performance issues:

Lost packets
Retransmits
Truncation
High delay
Excess resource utilization

Issues with dynamic routing


adjacencies

Fragmentation...

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

12

The Issue(s) with Fragmentation...

Breaks down large IP datagrams into smaller datagrams so


that it can be sent on a link with a smaller MTU

Fragmentation introduces more router load


Host is responsible for reassembling = more load on the host
If one fragment is missing, the entire original packet must be
resent

Only initial fragment contains upper-level protocol headers

Non-initial fragments will match only source/destination IP


information in ACLs
May be vulnerable to attack

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

13

The Bigger The Better?


DC02

Jumbo frame support is not widespread

North / South and inter-DC traffic should be


cautious of going above 1500 MTU
Never assume

P1.a
P1.b

DC01

Doesnt play well with real-time traffic

More prone to performance issues in terms


of RTT and latency

P1

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

14

Its Good Where It Makes Sense

East / West traffic; the busy parts


of the network

Storage traffic
Workload mobility
Isolated traffic

Private WAN where there is control


& visibility into the complete traffic
path

All stars need to align

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

15

Sizing Across Multiple Protocols

Making It All Flow...

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

17

NX-OS System QoS Policies


Policy Application Point

Policy Type

Function

system
qos

ingress
interface

qos

Define traffic
classification

queuing

Apply queuing
strategy; strict
priority, deficit
weight round
robing

network-qos

Define system
class: drop/nodrop, MTU,
buffer size

egress
interface

Match and define policy using the respective type of class-maps and
policy-maps
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

18

QoS Policy Types


*Platform dependent criteria
type qos

type queuing

type network-qos

Class-map

Class-map

Class-map

CoS, DSCP, ACL, IPP,

CoS, Protocol

CoS, DSCP

Protocol (TLV)

qos-group

qos-group

Policy-map

Policy-map

Policy-map

Marking, policing, mutation

MTU, No-drop (lossless Ethernet)

Set qos-group

Congestion control

Priority, queueing, buffering,


scheduling

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

19

Type Network-QoS Policies

One per system that is applied to the


Unified Crossbar and affects all L2
ports

Controls the shape of the traffic and


how it reacts

Exception is native FC interfaces

Ability for varying MTU values among


classes

Why use CoS?


Ethernet Wire
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

20

IPv4 Traffic

MTU may be determined based on:


Interface configuration <mtu x>
network-qos policy: mapped into classes based on CoS/QoS marking or qos-groups
Otherwise the default class value is used

L3 hop where packet exceeds MTU: fragment except when the DF-bit is present

Switched traffic exceeding the MTU is truncated or dropped


Consistent MTU scheme

Ingress
Module
Ingress
Module
Ingress
Module

Fabric

Egress
Module

Ingress
Module
Ingress
Module
Ingress
Module

Fabric

Egress
Module

BRKDCN-1100

Ingress
Module
Ingress
Module
Ingress
Module

Fabric

Egress
Module

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

21

IPv6

No fragmentation supported on routers!

Must rely on PMTUD, consistent MTU scheme, RA messages

Hosts are responsible for fragmentation


ICMPv6 Type2 messages must be allowed for PMTUD operation

MTU for IPv6 traffic may be set with the interface-level MTU command
or within a network-qos class

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

22

IPv6

NX-OS has one interface-level MTU command: mtu <size>

Influences IPv4 & IPv6 alike

Recommendation: have IPv6 routers send hosts the MTU that should be used
via Router Advertisements (ICMP packets)

Instructs the host to use an MTU of


1480 in packets it originates

N7706-A(config-if)# ipv6 nd mtu 1480


N7706-A(config-if)# sh ipv6 nd int e1/11
ICMPv6 ND Interfaces for VRF "default"
Ethernet3/1, Interface status: protocol-down/linkdown/admin-down
IPv6 address: 0dc3:0dc3:0000:0000:0218:baff:fed8:239d
ICMPv6 active timers:
Last Neighbor-Solicitation sent: never
Last Neighbor-Advertisement sent: never
Last Router-Advertisement sent:never
Next Router-Advertisement sent in: 0.000000
Router-Advertisement parameters:
Periodic interval: 200 to 600 seconds
Send "Managed Address Configuration" flag: false
Send "Other Stateful Configuration" flag: false
Send "Current Hop Limit" field: 64
Send "MTU" option value: 1480

BRKDCN-1100

RA

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

23

iSCSI

Block-level storage access over IP

Can co-exist with LAN infrastructure


TCP or PFC

iSCSI with Jumbo Frames

GOOD: Allows for entire SCSI data blocks in


packets = less buffer wait time and throughput is
consumed more efficiently
NOT SO GOOD: MTU mismatches severely hinder
performance with potential disconnects and data
corruption

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

24

iSCSI Best Practices


WAN

Intra-DC iSCSI
with existing LAN

Inter-DC iSCSI
Multiple L3 hops

Remote
iSCSI

LAN
iSCSI direct
attach to UCS FI

SAN

Use Jumbo Frames when there is control


over all transit devices

Best practice: isolate iSCSI traffic (i.e.


dedicated replication/backup VLANs), apply
separate CoS value
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

25

Sample VLAN-based iSCSI Policy


Reference 5672 Platform
IP ACL matches the well-known
iSCSI ports for the QoS policy

Sets the internal qos-group for


system class mapping and
applied in the VLAN database

Network-qos policy matches the


iSCSI traffic with the qos-group
and increases the MTU

ip access-list iSCSI-ACL
permit tcp 10.0.0.0/24 any eq 860
permit tcp 10.0.0.0/24 any eq 3260
!
class-map type qos match-all class-iSCSI
match access-group name iSCSI-ACL
!
policy-map type qos iSCSI-QOS-policy
class class-iSCSI
set qos-group 2
set cos 4
class class-default
!
vlan configuration 10
service-policy type qos input iSCSI-QOS-policy
!
class-map type network-qos iSCSI-nq
match qos-group 2
!
policy-map type network-qos jumbo-nq-policy
class type network-qos c-nq-iSCSI
mtu 9000
pause no-drop
class type network-qos c-nq-default
mtu 1500
!
system qos
service-policy type network-qos jumbo-nq-policy

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

26

NAS

File-level storage access over an IP


network

NFS or SMB
Storage can be shared simultaneously across
multiple computers

TCP or UDP based

May benefit from increased MTU but not


required

LAN

SAN

Mismatches result in latency and slow


performance
Similar to iSCSI principles, assign a consistent
CoS/QoS policy

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

27

Fibre Channel

Supported on Unified Port (UP) switches that include the Nexus 5500/5600/6000
series switches as well as the 2348UPQ FEX

Requires Jumbo Frames

Maximum receive size of 2148 bytes with headers


Maximum payload of 2112 bytes

MTU adjustments are hard-coded when interface is converted to fc and traffic is


identified/classified based on the physical interface it is received on
SOF

FC Header

4bytes

24bytes

Frame delimiters

SCSI Payload
Commands & Data
2112 bytes

CRC

EOF

4bytes 4bytes

Control information,
src/dst addresses
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

28

Fibre Channel over Ethernet (FCoE)

Still Fibre Channel

FC frame is encapsulated with an FCoE header and regular Ethernet headers


Classified based on its EtherType

CoS 3 is the standard FCoE QoS marking and will auto-associate with this class

ELP requires that rxbufsize match between peers, otherwise link fails
2180 Bytes
Ethernet
Header
12bytes
+ 4bytes
.1Q tag

FCoE
Header

FC Header

16bytes

24bytes

SCSI Payload
Commands & Data
2112 bytes

CRC

EOF

FCS
4bytes 4bytes 4bytes

FCoE identified
with Ethertype
0x8906
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

29

FCoE Among Platforms

N56128-A# sh policy-map system


Type network-qos policy-maps
============================
policy-map type network-qos jumbo+_fcoe-nq-policy
class type network-qos class-fcoe
pause no-drop
mtu 2158
class type network-qos class-default
mtu 9000

N7706-Admin-A# sh policy-map system


Type network-qos policy-maps
============================
policy-map type network-qos jumbo-nq-7e-4q8q-policy template 7e-4q8q
class type network-qos c-nq-7e-4q8q-drop
congestion-control tail-drop
mtu 9000
class type network-qos c-nq-7e-4q8q-ndrop-fcoe
pause
mtu 2112

Header length is internally added and


implemented in hardware with fixed sizes
2112 + 18 (eth_hdr&CRC) + 28
(FC_hdr&CRC) = 2158 bytes

N7K: Must use a dedicated Storage


VDC and a network-qos policy

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

30

Fibre Channel over IP (FCIP)


FC SAN
FC SAN
FCIP
end-point

FCIP
end-point

Long distance SAN extension between FCIP end-points

Available on MDS platforms

Leverages TCP for resilience and congestion control

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

31

FCIP Frame
Ethernet
Header

14

IP
Header

TCP
Header

TCP
Opts

FCIP
Header

S
O
F

EISL
Hdr

Opt
Hdr

20

20

12

28

0-16

FCIP Overhead: 98 Bytes


Includes 94 byte header
& 4 byte CRC

FC Frame

CRC

Maximum FC frame size:


2148 bytes

FCIP will segment and reassemble FC frames if MTU exceeds

Enabling Jumbo frames across the path will increase performance

DF-bit is set between FCIP end-points


BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

32

FCIP MTU & Discovery

FCIP MTU
recommendation is
2300 bytes

Path MTU
Discovery is
enabled by default
FCIP end-points

Increase MTU on physical


interface; will also increase
on FCIP interface

Any MTU changes across


path will be discovered
within 3600 secs

MDS-A(config)# feature fcip


MDS-A(config)# int gi1/1
MDS-A(config-if)# ip address 10.1.1.1/24
MDS-A(config-if)# mtu 2300
MDS-A(config)# fcip profile 10
MDS-A(config)# ip address 10.1.1.1
MDS-A(config)# int fcip 10
MDS-A(config-if)# use-profile 10
MDS-A(config-if)# peer-info ipaddr10.1.1.2
MDS-A(config-if)# no shut
MDS-A(config-if)# show fcip profile 10
FCIP Profile 10
Internet Address is 10.1.1.1 (interface GigabitEthernet1/1)
Tunnels Using this Profile: fcip10
Listen Port is 3225
TCP parameters
SACK is enabled
PMTU discovery is enabled, reset timeout is 3600 sec
Keep alive is 60 sec
Minimum retransmission timeout is 200 ms
Maximum number of re-transmissions is 4
Send buffer size is 16384 KB
Maximum allowed bandwidth is 5000000 kbps
Minimum available bandwidth is 4000000 kbps
Configured round trip time is 1000 usec
Congestion window monitoring is enabled, burst size is 50 KB
Auto jitter detection is enabled

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

33

MTU Interaction with Data Center


Technologies

vPC
Overview
vPC Logical View

vPC Physical View

STP

7K-1
7K-1

7K-2

7K-1

SW1

7K-2

7K-2

SW1

SW1

All links forwarding: no blocked ports, increased bandwidth, load sharing

High availability & redundancy

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

35

vPC
MTU Areas of Interest
vPC Domain

Peer-Link

vPC Peer-Keepalive

Internally programmed for 9216+


padding
Extra padding is due to Cisco Data
Center Ethernet Protocol (CDCE)

vPC Peer-Link

vPC Member
Ports

MAC-in-MAC
EtherType 0x8903

Overhead auto-accounted for

vPC

vPC

vPC

Downstream device

Should have same MTU as vPC


Member Ports

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

36

vPC
Member Port MTU

Applies to switches where interface-level MTU config is used

MTU config must match on vPC port-channel member interfaces between


vPC peers

Physical interfaces auto-inherit

Mismatch causes forwarding interruption for that specific vPC (Type-1)


N7706-A# sh vpc consistency-parameters inter po 11
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name
------------mode
Speed
Duplex
Port Mode
Native Vlan
MTU
Admin port mode

Type
---1
1
1
1
1
1
1

Local Value
---------------------active
10000 Mb/s
full
trunk
1
9000
trunk
BRKDCN-1100

Peer Value
----------------------active
10000 Mb/s
full
trunk
1
9000
trunk

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

37

vPC
Global Consistency Check

Applies to Nexus switches with network-qos policies

vPC still operates despite MTU mismatch within the network-qos policy
(Type-2)

*No reason for it not to match


Nexus56128P-A# sh vpc consistency-parameters global

qos-group 0 (default)

qos-group 1 (FCoE)

Legend:
Type 1 : vPC will be suspended in case of mismatch
Name
------------QoS

Type
---2

Network QoS (MTU)

Network Qos (Pause)

Local Value
---------------------([], [3], [], [], [],
[])
(9216, 2196, 0, 0, 0,
0)
(F, T, F, F, F, F)

BRKDCN-1100

Peer Value
----------------------([], [3], [], [], [],
[])
(9216, 2196, 0, 0, 0,
0)
(F, T, F, F, F, F)

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

38

Why Overlays

Encapsulated Traffic

Virtual Switch

Virtual Switch

Resilient Underlay

Overlay Virtual Network

High capacity interconnects

Scalable

Transit devices are agnostic

Flexible

Must be capable of handling


overhead

L2 or L3

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

39

FabricPath
Benefits

S100

S10

S20

S30

S40

S200

S300

S400

S500

FabricPath

S600

Maintains Layer 2 connectivity without the use of STP

Efficient while utilizing ECMP

Enables flexibility within the Data Center and is highly scalable

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

40

FabricPath
Control Plane Operation

Utilizes IS-IS to dictate the forwarding topology

Essentially L3 routing to the L2 bridged domain


Computes shortest path within the FabricPath domain
Supports the use of equal-cost paths

No L2 IS-IS configuration needed but is tunable

IS-IS LSPs have an MTU of 1492 bytes


Can adjust with lsp-mtu <size> command but no documented performance benefit
since it is strictly control traffic
Recommend to leave at default

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

41

FabricPath
Data Plane

All fields retained, new CRC


S20

S10

OutDA

OutSA

FPTag

DMAC

SMAC

S100

16 bytes total overhead


within FP Domain

DMAC

.1Q
S200

SMAC

.1Q

Etype

S30

Payload

S300

Etype

CRC
S400

Payload

S40

S500

S600

CRC

FP header removed as
frame returns to CE domain
Classical Ethernet Frame
Ingress into FP Domain

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

42

FabricPath
Data Plane

The 16 bytes of overhead is automatically accommodated by the ASICs when


the interface is converted to switchport mode fabricpath

Same EtherType (CDCE) as on vPC Peer-Link

MTU configurations and adjustments on the platform are reflected in the


FabricPath forwarding domain:
N7706-Spine-A# show fabricpath isis interface brief
Fabricpath IS-IS domain: default
Interface
Type Idx
State
Circuit
MTU Metric Priority Adjs/AdjsUp
-------------------------------------------------------------------------------port-channel10 P2P
1
Up/Ready
0x01/L1
9000 10
64
1/1

port-channel20 P2P

Up/Ready

0x01/L1

9000

10

64

1/1

port-channel30 P2P

Up/Ready

0x01/L1

9000

10

64

1/1

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

43

OTV

OTV

L3 DCI

Overview

OTV Edge

OTV Edge
OTV Edge

OTV Edge

OTV Inline

LAN Extension over any network

Core

Dark Fiber, MPLS, IP...


Multi-data center scalability with
ability for workload mobility

Network-based overlay with


stateless tunneling

Core

Core

Core

West DC

East DC

VLANs 100-109

VLANs 100-109

OTV Join
Interfaces

Core Transit

Overlay Interface

OTV
Edge

Core Device

OTV on a stick
Aggregation Device
Internal Interfaces

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

44

OTV
Packet Types
Packet Type

Purpose

IP Length

ISIS Hello;
Unicast

Establish OTV
Adjacency

1450 Bytes

ISIS Hello;
Multicast*

Establish OTV
Adjacency

1442 Bytes

OTV Header

Added to all OTV


Data traffic

Pkt_len+42 Bytes

*Recommended
OTV control

In order to forward a standard 1500 byte packet from a host application,


ensure transport supports at least a 1542 byte MTU end-to-end

Alternatively the host MTU could be decreased to accommodate overhead


BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

45

OTV
Data Packet Format

MAC-in-IP

802.1Q tag is removed from original L2


header
VLAN ID copied to OTV Shim
802.1P bits copied to new IP Header and
OTV Shim

Fragmentation is not supported due to L2


presumption and hiding of L3 network

OTV
Overhead

DF bit is set on all OTV control and data


traffic between Edge devices

New L2 Header
14 Bytes
New IP Header
20 Bytes
OTV Shim
8 Bytes
Original L2 Header
14 Bytes

Original IP Packet
Payload

Original
Frame

No PMTUD either
L2 FCS
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

46

OTV

MTU 1542 bytes

Interface Config
7K-A-CORE(config-if)#
7K-A-CORE(config-if)#
7K-A-CORE(config-if)#
7K-A-CORE(config-if)#
7K-A-CORE(config-if)#
7K-A-CORE(config-if)#

int po10
description OTV Internal Int
mtu 1542
int e1/1
description Core Uplink
mtu 1542

OTV Edge
Device

Core Device

Aggregation Device
7K-A-OTV(config-if)#
7K-A-OTV(config-if)#
7K-A-OTV(config-if)#
7K-A-OTV(config-if)#
7K-A-OTV(config-if)#

int po10
description Join Interface
mtu 1542
int overlay1
otv join-interface po10

OTV Internal Interface


OTV Join Interface

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

47

VXLAN
What Does it Accomplish?

Flexible host-based overlay with


mobility across data centers and
clouds

Segmentation and Scale

Underlay

Stretches the L2 segment ID field to 24


bits = up to 16 million L2 segments
Extends L2 segments over a L3
boundary VXLAN encapsulates L2
frames in an IP/UDP header

VXLAN VNI 100

Allows for all links forwarding via L3


ECMP

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

48

VXLAN
1500 byte packet

Overhead and Transport

14 byte inner-Ethernet header

8 byte VXLAN ID
8 byte UDP header
VXLAN Encapsulation

20 byte IP header

50 Bytes (possibly 54)

1550 byte minimum VXLAN MTU

MAC Header

IP Header

UDP Hdr

14 bytes (4 bytes optional)_

20 Bytes

8 Bytes

Outer
MAC
DA

Outer
MAC
SA

Transport MAC
addresses

Outer
802.1Q

Outer
IP
DA

Outer
IP
SA

Outer
UDP

VXLAN
ID
(8 bytes)

Inner
MAC
DA

Inner
MAC
SA

VM MAC
addresses
Transport
VLAN
(Optional)

IP addresses
of the VTEPs

UDP destination
port 4789 identifies
the packet as
VXLAN traffic;
source UDP is a
hash of inner frame

VNI Field
identifies the L2
segment

BRKDCN-1100

Ethernet
Payload

CRC

The original frame; at


default 1514 bytes or
1518 bytes

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

49

VXLAN
Operation

Virtual Tunnel End Points


(VTEP) perform VXLAN
encapsulation / de-capsulation

Add 50 bytes to transit links to


support MTU of 1550 bytes

IPv4 Transport

VTEP

Hypervisor

VTEP

Alternatively MTU could be


decreased by 50 bytes on
servers

VTEP

LAN segment for bridging


& IP interface known in the
transport network (usually
a loopback)

When 1000v is used as VTEP,


it can assist with MTU issues

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

50

Ins and Outs of MTU Related


Configurations

Logical Flow
Operating System

Compute

Switching
Storage

OS Kernel

Driver

Adapter

IOM

FI

1000v

Nexus
Switch

IP

MTU

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

52

Influencing Traffic Size


IOS vs. NX-OS

IOS

NX-OS

ip mtu

mtu

mtu

network-qos policy

ip tcp adjust-mss
system mtu
...

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

53

Nexus 7000 & 7700 Series

Store-and-forward switch

May use interface-level MTU


commands or network-qos policy
depending on interface

Features such as FCoE and FEX


require network-qos policies

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

54

Interface-Level MTU
Layer 2

Jumbo MTU is supported on the


switch by default but needs to be
enabled

Interface-level must use the


standard default MTU of 1500 or
match the value specified in
system jumbomtu command

N7706-A(config)# sh run all | i jumbomtu


system jumbomtu 9216
N7706-A(config)# sh int e1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address:
Description: 5K-A 1/4
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
N7706-A(config)# int e1/1
N7706-A(config-if)# mtu 9216
N7706-A(config)# sh int e1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address:
Description: 5K-A 1/4
MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255

Applies for port-channels as well


Nexus7706-A(config)# int e1/1
Nexus7706-A(config-if)# mtu 9000
ERROR: Ethernet1/1: MTU on L2 interfaces can only be set to default or system-jumboMTU

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

55

N7K network-qos Policies

Alternative MTU configuration is via the


network-qos policy in the Admin VDC

Applies to L2 traffic in all VDCs


Class-maps match on CoS and protocol
Disruptive

Default policy cannot be modified; copy


& modify a predefined template or
create one that conforms to a template

Any interface-level MTU will take


precedence

Example policy with 7 drop


classes & 1 no-drop (FCoE)
N7706-A# qos copy policy-map type network-qos default-nq-7e-4q8q-policy
prefix CUSTOMN7706-A# conf
N7706-A(config)#policy-map type network-qos CUSTOM-nq-7e-4q8q-policy
N7706-A(config-pmap-nqos)# class type network-qos c-nq-7e-4q8q-drop
N7706A(config-pmap-nqos-c)# mtu 9216
N7706A(config-pmap-nqos-c)# class type network-qos c-nq-7e-4q8q-ndropfcoe
N7706A(config-pmap-nqos-c)# mtu 2112
N7706A(config-pmap-nqos-c)# system qos
N7706-A(config-sys-qos)# service-policy type CUSTOM-nq-7e-4q8q-policy
!
N7706-A# sh policy-map system type network-qos
policy-map type network-qos CUSTOM-nq-7e-4q8q-policy template 7e-4q8q
class type network-qos c-nq-7e-4q8q-drop
congestion-control tail-drop
mtu 9216
class type network-qos c-nq-7e-4q8q-ndrop-fcoe
pause
mtu 2112

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

56

N7K network-qos Templates

CoS 6 & 7 reserved for


control traffic (nonconfigurable)
CoS 3 reserved for
FCoE

Template

Drop CoS

Priority

NoDrop CoS

default-nq8e-policy

0,1,2,3,4,5,6,7

5,6,7

default-nq7e-policy

0,1,2,4,5,6,7

5,6,7

default-nq6e-policy

0,1,2,5,6,7

5,6,7

3,4

default-nq4e-policy

0,5,6,7

5,6,7

1,2,3,4

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

57

Nexus 7K
Layer 3 Interfaces

Configured directly on any L3


interface: L3 ports, subinterfaces,
SVIs, port-channels, port-channel
subinterfaces

Parent port-channel interface must have


an equal or greater size

Values can differentiate among L3


interfaces with up to 64 settings per
module

N7706-A(config)# int po10


N7706-A(config-if)# mtu 9000
N7706-A(config-if)# int po10.100
N7706-A(config-if)# mtu 2200
N7706-A(config-if)# sh int po10
port-channel10 is up
admin state is up
Hardware: Port-Channel, address:
Description: 7K-A Agg
MTU 9000 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is routed
N7706-A(config-if)# sh int po10.100
port-channel10.100 is up
admin state is up, [parent interface is port-channel10]
Hardware: Port-Channel, address:
Internet Address is 10.1.1.1/30
MTU 2200 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 100, medium is broadcast
Port mode is routed

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

58

Nexus 7K Verification & Troubleshooting

Software verification of port


interface MTU

N7706-A# sh system internal ethpm info interface po10 | i


MTU
medium (broadcast), snmp trap(on), MTU(9000),

N7706-A# sh system internal qos network-qos hw-config | i MTU|slot|CoS


slot 1
UF Configs for CoS 0
MTU
= 1500 [FCoE: No]
UF Configs for CoS 1
MTU
= 1500 [FCoE: No]
UF Configs for CoS 2
MTU
= 1500 [FCoE: No]
UF Configs for CoS 3
MTU
= 2112 [FCoE: Yes]
UF Configs for CoS 4
MTU
= 1500 [FCoE: No]
UF Configs for CoS 5
Inidicates whether port is
MTU
= 1500 [FCoE: No]
using intf level MTU or
UF Configs for CoS 6
inheriting the MTU from the
MTU
= 1500 [FCoE: No]
UF Configs for CoS 7
Virtual Link
MTU
= 1500 [FCoE: No]
Interface
Config Oper(VLs) MTU (value)
Ethernet1/1 Auto Off(0) Port-MTU (9000)
Ethernet1/2 Auto Off(0) Port-MTU (9000)
Ethernet1/3 Auto Off(0) Port-MTU (9000)
Ethernet1/4 Auto Off(0) VL-MTU
(-)
Ethernet1/5 Auto Off(0) Port-MTU (9000)
Ethernet1/6 Auto Off(0) Port-MTU (9000)
Ethernet1/7 Auto Off(0) Port-MTU (9000)
Ethernet1/8 Auto Off(0) Port-MTU (9000)
Ethernet1/9 Auto Off(0) Port-MTU (9000)
Ethernet1/10 Auto Off(0) Port-MTU (9000)
Ethernet1/11 Auto Off(0) Port-MTU (9000)
Ethernet1/12 Auto Off(0) Port-MTU (9000)
Ethernet1/13 Auto Off(0) VL-MTU
(-)
Ethernet1/14 Auto Off(0) VL-MTU
(-)
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

59

Nexus 7K Verification & Troubleshooting


A configurable rate-limit for packets needing to be
fragmented may be configured; by default it is 500 pps
N7706-A# sh hardware rate-limiter layer-3 mtu
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
rl-1: STP and Fabricpath-ISIS
rl-2: L3-ISIS and OTV-ISIS
rl-3: UDLD, LACP, CDP and LLDP
rl-4: Q-in-Q and ARP request
rl-5: IGMP, NTP, DHCP-Snoop, Port-Security, Mgmt and Copy traffic

Equals 9038 in decimal


(internal header
accommodation)
N7706-A-Admin# attach module 1
Attaching to module 1 ...
To exit type exit, to abort type $.
module-1# sh hardware internal mac port 1 qos | i MTU
Port MTU (No dropping action): 9038

module-1# sh hardware internal mac port 1 sw-state | b MTU


# Port MTU #
port adj_mtu=0x26 last_mtu=0x2316 qos_mtu=0x0 pm_mtu=0x234e
vl0 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl1 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl2 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl3 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl4 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl5 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl6 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl7 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600

Module: 1
Rate-limiter PG Multiplier: 1.00
R-L Class
Config
Allowed
Dropped
Total
+------------------+--------+---------------+---------------+-----------------+
L3 mtu
500
106
0
106

Any packets exceeding the ratelimit will be dropped and counted

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

60

Nexus 9000 (Standalone)


MTU Overview

Similar to 7K

Configure MTU for L3 interfaces


directly on the interface
Interface-level MTU for L2 interfaces is
supported (1500 or system
jumbomtu <size>)

System policies (network-qos) may


also be used to apply to all L2
interfaces

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

61

Nexus 5500 & 5600


Architecture Overview

Switching paradigms:

Cut-through switch [default for 10G]: frames exceeding the


MTU are truncated and result in eventual CRC errors
Store-and-forward: frames are dropped when MTU is
exceeded

Layer 3 capabilities are integrated at line rate for 5600


series and are available with L3 module for 5500 series

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

62

Nexus 5500 & 5600


Switching Modes

When 10G
Ethernet is used
ingress, cutthrough switching
is used by default

Src Int

Dst Int

Switching Mode

10G Ethernet

10G Ethernet

Cut-through

10G Ethernet

1G Ethernet

Cut-through

1G Ethernet

1G Ethernet

Store-and-Forward

1G Ethernet

10G Ethernet

Store-and-Forward

FCoE

Fibre Channel

Cut-through

Fibre Channel

FCoE

Store-and-Forward

Fibre Channel

Fibre Channel

Store-and-Forward

FCoE

FCoE

Cut-through

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

63

Nexus 5500 & 5600


Layer 2 Interfaces

Must use a nework-qos policy-map

Per interface MTU not supported


One system-wide policy

qos-groups uniquely identify the system


classes internally

MTU may be configured


on a per class basis

5 configurable, 1 default class and 2 reserved for


control traffic

System defaults
system jumbomtu 9216; configurable from
2158 to 9216 bytes
Default network-qos policy cannot be modified

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

64

Nexus 5500 & 5600


Layer 2 MTU Configuration
N5672-A(config)# policy-map type network-qos jumbo-nq-policy
N5672-A(config-pmap-nq)# class type network-qos class-fcoe
Both commands
N5672-A(config-pmap-nq-c)# pause no-drop
required for
N5672-A(config-pmap-nq-c)# mtu 2158
FCoE class
N5672-A(config-pmap-nq-c)# class type network-qos class-default
N5672-A(config-pmap-nq-c)# mtu 9000
N5672-A(config)# system qos
N5672-A(config-sys-qos)# service-policy type network-qos jumbo-nq-policy
N5672-A# sh policy-map system
Type network-qos policy-maps
===============================
policy-map type network-qos jumbo-nq-policy
class type network-qos class-fcoe
match qos-group 1
pause no-drop
mtu 2158
class type network-qos class-default
match qos-group 0
mtu 9000
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

65

Nexus 5500 & 5600


L2 MTU verification
N5672-A# sh int e1/37
Ethernet1/37 is up
Dedicated Interface
Hardware: 1000/10000 Ethernet, address:
Description: Dell VRTX Eth8/1
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
<output omitted>
30 seconds input rate 93872 bits/sec, 38 packets/sec
30 seconds output rate 170712 bits/sec, 101 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 80.43 Kbps, 15 pps; output rate 154.54
Kbps, 78 pps
RX
38255265 unicast packets 256631 multicast
packets 418643 broadcast packets
38930539 input packets 23961814506 bytes
11702772 jumbo packets 0 storm suppression bytes
<output omitted>
TX
71224821 unicast packets 89774097 multicast
packets 4865803 broadcast packets
165864721 output packets 84561892595 bytes
45431246 jumbo packets
<output omitted>

N5672-A# sh queuing int e1/37


Ethernet1/37 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0
WRR
50
1
WRR
50
RX Queuing
qos-group 0
q-size: 100160, HW MTU: 9000 (9000 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Pkts received over the port
: 449082142
Ucast pkts sent to the cross-bar
: 448848703
Mcast pkts sent to the cross-bar
: 233439
Ucast pkts received from the cross-bar : 244633695
Pkts sent to the port
: 317668644
Pkts discarded on ingress
: 0
Per-priority-pause status
: Rx (Inactive), Tx (Inactive)
qos-group 1
q-size: 165120, HW MTU: 2158 (2158 configured)
drop-type: no-drop, xon: 62720, xoff: 88320
Statistics:
Pkts received over the port
: 3446851283
Ucast pkts sent to the cross-bar
: 3446902243
Mcast pkts sent to the cross-bar
: 0
Ucast pkts received from the cross-bar : 2610115540
Pkts sent to the port
: 2610115564
Pkts discarded on ingress
: 0
Per-priority-pause status
: Rx (Active), Tx (Active)
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

66

Nexus 5500 & 5600 Troubleshooting

Using cut-through forwarding, frame is passed to egress ASIC once DA is


parsed

5K is receiving 9216 byte


frames while configured for
only 1500 MTU

Excellent command for


seeing packet sizes / types
on an interface

N56128-A# sh hardware internal bigsur port e1/48 counters rx

Port 10gb1/48 MAC statistics:


MAC Statistic
| Value
| Diff
| Rate
|
-------------------------------+-----------------------------------+----------------------+--------------+
<snip>
RX_PKT_SIZE_IS_8192_TO_9216
|181538
|181538
|0
N56128-A# sh hardware internal bigsur all-ports

Determine which bigsur ASIC


the interface belongs to...

Switch is detecting
MTU violations;
packets are truncated

Bigsur Port Info:


Port
|asic|inst|inst|
name
|idx |slot|asic|eport|logi|flag|adm|opr|if_index|diag|ucVer
---------+----+----+----+-----+----+----+---+---+--------+----+----<snip>
xgb1/48 |3
|0
|3
|11 p |47 |b3 |en |up |1a02f000|pass| 0.00
N56128-A# sh hardware internal bigsur asic 3 counters interrupt | i mtu
Slot 0 Bigsur3 interrupt statistics:
Interrupt name
|Count
|ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+---<snip>
big_bmin_cl1_INT_p1_err_ig_mtu_vio
|30
|0
|30
|0

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

67

Nexus 5500 & 5600 Troubleshooting

On the egress 5K interface;


output errors are also
shown in show interface

N56128-A# sh hardware internal bigsur port e1/4 counters rx


Port 10gb1/4 MAC statistics:
MAC Statistic
| Value
| Diff
| Rate
|
-------------------------------+-----------------------------------+----------------------+--------------+
<snip>
TX_PKT_FRAME ERROR
|181538
|181538
|0
N56128-A# sh hardware internal bigsur asic 0 counters interrupt | i crc

Frames are CRC stomped

Slot 0 Bigsur0 interrupt statistics:


Interrupt name
|Count
|ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+---<snip>
big_fwe_psr1_P2_INT_pkt_err_eth_crc_stomp
|30
|0
|30
|0

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

68

Nexus 5500 & 5600


L3 Interfaces

Set MTU directly on the interface

Use mtu <size> syntax under the interface (physical, SVI,


Port-Channel)

Can only support an MTU of up to 9192 on L3 interfaces

Caused by 24 byte switch internal header


Implies that other devices should be adjusted to
accommodate if L3 is needed on 5K

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

69

Nexus 3000, 3100, 3500 Series

Nexus 3000

Nexus 3100, 3500

Per-port MTU configuration


supported for L2 and L3 interfaces

May use to network-qos policy to


affect all ports

network-qos policy must be used

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

70

Nexus 2200 & 2300FEX


Architecture Overview

Jumbo frame support is dependent


on the parent switch configuration

Auto-inherit of network-qos policy

No local switching on FEX

FEX links use a VNTag (6 bytes) & an


802.1Q tag (4 bytes)
Programed when NIFs are configured

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

71

Nexus 2200 & 2300FEX


7K Parent

6.2(2)+:All FEX MTU controlled by


network-qos policy

Prior releases only MTU modification was


needed on FPC

Create a new network-qos policy in the


Default or Admin VDC

Changes are automatically pushed


down to the FEX

Example enables jumbo


frames for all traffic classes
N7706-Admin-A(config)# class-map type network-qos
match-any class-nq-8e
N7706-Admin-A(config-cmap-nqos)# match cos 0-7
N7706-Admin-A(config)# policy-map type network-qos
jumbo-nq-8e template 8e-4q8q
N7706-Admin-A(config-pmap-nqos)# class-map type
network-qos class-nq-8e
N7706-Admin-A(config-pmap-nqos-c)# mtu 9000
N7706-Admin-A(config-pmap-nqos-c)# congestion taildrop
N7706-Admin-A(config)# system qos
N7706-Admin-A(config-sys-qos)# service-policy type
network-qos jumbo-nq-8e

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

72

Nexus 1000v
VMware (N1K 5.x)

system jumbomtu 9000 is a fixed global setting

For jumbo frame forwarding, MTU config needed on eth port-profiles OR


interfaces
N1kv-VSM-A(config)# port-profile type eth Uplink_PP
N1kv-VSM-A(config-port-prof)# mtu 9000

Nexus1000v-VSM-A(config)# int e3/1


Nexus1000v-VSM-A(config-if)# mtu 9000

Confirm on ESXi host:

[root@UCSC240:~] esxcfg-nics -l
Name
PCI
Driver
vmnic0 0000:02:00.0 igb
vmnic1 0000:02:00.1 igb

Link Speed
Up
1000Mbps
Up
1000Mbps

Duplex MAC Address


MTU
Full
a4:93:4c:aa:fb:6e 9000
Full
a4:93:4c:aa:fb:6f 9000

Description
Intel Corporation I350 Gigabit Network Connection
Intel Corporation I350 Gigabit Network Connection

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

73

Nexus 1000v
Microsoft Hyper-V

No management over MTU settings


within the virtual modules

Physical adapters added to 1000v


should have desired MTU

Modify via the MS adapter properties,


UCSM, or ILO where applicable
Value should not be changed after NIC is
added to the switch

MTU value not reflected on VSM/VEM


interfaces; verify on host with
.\vemcmd.exe show attach info

---------------------------------------------LTL:
17
---------------------------------------------Port ID:
4
NIC Index:
2
NIC Instance ID: {7C3A395E-ADDF-41B8-BE39D05BC73C4699}
MAC address:
a0:36:9f:6a:c8:92
Port profile:
UPLINK-PP
Port profile GUID: 6AF1BB76-F87A-4EAB-94808420AFB6CD02
Network GUID:
VM/NIC name:
Intel(R) Ethernet 10G 2P X520
Adapter
VM UUID:
NIC UUID:
MTU:
9000
Link state:
UP
Duplex:
Full
Tx speed:
1410065408
Rx speed:
1410065408
Autoneg:
Enabled
Link Params pending: No
Speed Capability 0x10
Duplex Capability 0x4

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

74

UCS Manager
Fabric Interconnect Processing
Host

Ingress
Policing
Violation =
truncate or drop
Threshold reached =
-Tail drop for drop classes
-Pause for lossless

MTU Check

Per-class
buffer monitor

8 multicast
queues

Egress Traffic

Traffic
Classification

Ingress Traffic

Trusted CoS
L2/L3/L4 Details

Host

Marking

CoS

Egress
Scheduler

Strict Priority or
DWRR Scheduler

Egress
Queues

Unicast VOQ
(8 per egress interface)

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

75

UCS Manager
QoS System Classes

Match criteria

Manually enter desired MTU


for each class
normal: 1500 bytes

Auto-created / hard-set with


vHBA creation;
fc: 2158 bytes
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

76

UCS Manager
Assigning to vNICs

Specify the MTU on the vNIC;


informs the host network adapter of
MTU value to be used
vNICs can be assigned
specific priority; otherwise
will default to Best Effort

9000

6248-FI-A# scope service-profile server 1/1


6248-FI-A /org/service-profile# show vnic
vNIC:
Name
Fabric ID Dynamic MAC Addr
Virtualization Preference
-------------- ---------- ----------------- ------------------------1-host-vNIC-A
A
00:25:B5:0A:00:01 NONE
1-host-vNIC-B
B
00:25:B5:0B:00:01 NONE
6248-FI-A /org/service-profile# show vnic 1-host-eth-1 detail
vNIC:
<output omitted>
Host Interface Ethernet MTU: 9000

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

77

End-to-End Testing

Verification across platforms


Linux: ping 192.168.1.5 M do s 8972 OR tracepath
ESXi: vmkping d s 8972 192.168.1.5
Windows: mturoute OR

Subtract the IP header &


ICMP header (28 bytes)
from the ping size

Be aware of default CoPP policy that rate-limits jumbo ICMP packets


destined to the local Nexus switch
BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

78

Summary

In Conclusion

Not all traffic is created equal

Know the flow

Align the stars

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

80

Complete Your Online Session Evaluation

Give us your feedback to be


entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.

Complete your session surveys


through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online

BRKDCN-1100

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

81

Continue Your Education

Demos in the Cisco campus

Walk-in Self-Paced Labs

Lunch & Learn

Meet the Engineer 1:1 meetings

Related sessions

Presentation ID

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

82

Thank you