ACI Fabric Discovery Guide

How to Use This Lab Guide
The purpose of this lab guide is to help you get the best value from your training session. In
order to ensure that relevant information is readily apparent to you, the following formatting
standards are used throughout this document.
General Information:
All Instructions are separated from the named object/button/selection with a colon
(:).
All items to be clicked on, typed, expanded, etc., are BOLD AND IN ALL CAPS
(unless case sensitive).
All student tenant numbers are indicated with XX – as in TXX-CDP-ENABLE –

which should result in a configured CDP policy for Tenant 1 as: T01-CDP-
ENABLE.
Navigate To: The Navigate To instruction is used when the guide is requesting that
you locate a page or section of the user interface. All Navigate To instructions are
annotated as follows:
Navigate to: FABRIC → ACCESS POLICIES
Arrow icons (→) indicate that the subsequent section is a child object of or can be
accessed from the previous section.
Expand: The Expand instruction is used when the guide is requesting that you expand
an object in the user interface. This will be used to display more information or
subsequent child sections. All Expand instructions are annotated as follows:
Expand: POOLS
Right-Click: The Right Click instruction is used to request that you right click on an
object to access a menu or perform an action. All Right Click instructions are
Right-Click: VLAN
Select: The Select instruction is used to request that you select an option in the user
interface. All Select instructions are annotated as follows:
Select: CREATE VLAN POOL
Name: The Name instruction is used to provide you the name that you should use
for a configurable object in the user interface or CLI. All Name instructions are
Name: TXX-VLANPool
Notes from the Field: While the lab guides are certainly intended to be useful and practical
to real world deployments, it is not always possible to ensure that the configurations in the labs
are “best practice.” Where applicable the Lumos flame icons (surrounding this call out) identify
Lumos: Be Brilliant. 1
what real world best practices would be for a configuration or clarify why the lab is configured
in a certain way.
ACI Fabric Discovery
Table of Contents:
Task 1: Gathering Information Needed for Setup
Task 2: Initial ACI Fabric Configuration
Task 3: Fabric Pod and Access Policies
Task 4: Out-of-Band (OOB) Management
Task 5: APIC GUI Overview
Screenshots are provided to guide you through each step. These screenshots are
based on the lab using Tenant #1 as a visual aide. In many cases, you will need to
replace the information in these screenshots with your Student/Tenant # and/or
information from a particular reference table.
Activity Objective
In this activity, students will be shown the process for the APIC startup process and initial
fabric configuration. As the ACI simulators no longer provide a full simulation of this feature,
this lab will provide a walk though illustrating the process and steps needed to complete the
initial configuration of the APIC, the ACI fabric discovery process, and a few steps needed to
ensure proper communication that are not presented in the student labs (NTP, BGP Route
Reflectors, vPC Protection Groups).
Required Resources
These are the resources and equipment required to complete this activity:
• Workstation with Internet access.
• Access to Lumos RDP server with the proper credentials.
• Access to the Lumos ACI fabric through the RDP server using information provided
below.
• Credentials for the APIC
◦ Username: admin
◦ Password: lumos123
APIC IP Address
https://10.203.254.24
RDP Connection
rdp.fab3.lumoscloud.com
Tasks
Task 1: Gathering Information Needed for Setup
In this task, students will review the information needed by the fabric installation and cover the
APIC initial configuration process. This process cannot be performed by the students at this
time but is provided as a reference guide.
The first step that needs to be completed is to gather the information you will need for the
installation. It is important to have this information gathered ahead of time as some of this
information can only be input while running the fabric setup. For example, if the TEP address
pool needs to be changed the fabric will need to be wiped and the setup process run again.
Other items like the Infra VLAN selected can have an impact on how traffic connects to the
legacy environment. These items will be called out in the sections below where needed.
Items in BOLD can only be changed by wiping the fabric:
Fabric name: [ACI Fabric 1]
The fabric name is used by all devices during the discovery process. Some
design considerations are worth mentioning here: Will there ever be more than
one fabric? Different fabrics in multiple DCs? Is there a naming convention that
makes more sense, for example, most enterprises have site codes that could be
used to more easily identify fabric location or purpose?
Fabric ID: [1]
Relevant for multi-site if using the same fabric name but different fabric IDs for
sites.
Number of controllers: [3]
You must have at least 3 APICs in a cluster. Up to 5 APICs are currently

supported with multi-pod.
Pod ID: [1]
Used for multi-pod.
Standby controller: [Y]
Standby controllers are used as a swappable spare but are not active in the
cluster until promoted.
Standby Controller ID:
Generally use 4 or 6.
Controller ID: [1]

Controller ID 1 will boot first and do discovery. Controller 2 won't initialize until
controller 1 is up. Controller 3 won't initialize until controllers 1 and 2 are up.
Controller name: [apic1]
Hostname for APIC. This is what will be displayed at the CLI prompt.
TEP address pool: [10.0.0.0/16]
This value is for the infrastructure virtual routing and forwarding (VRF) only. It
needs to be large enough to handle all devices that will need a VTEP address
assigned. In certain design scenarios, this can be a very large number of
addresses. This subnet should not overlap with any other routed subnets in the
network. The minimum supported subnet for a 3 APIC cluster is /22.
INFRA VLAN: [4]

Reserve this VLAN for APIC use only. The infrastructure VLAN ID must not be
used elsewhere in the environment and must not overlap with any other reserved
VLANs on other platforms (for example the reserved vlan range on Nexus 7K). If
this does conflict, L2 extension to those devices may not work or the reserved
VLAN may have to be changed. It is a common best practice to use VLAN 3967
for the Infra VLAN -- this is the last assignable VLAN on most Nexus devices
(3968-4094 are system reserved VLANs).
GIPO multicast pool: [225.0.0.0/15]

IP range used for fabric multicast. Valid ranges are from 225.0.0.0/15 to
231.254.0.0/15, prefix length must be 15 (128k IPs).
Enable IPv6 management: [N] IPv4 management IP:
IP address used for OOB management interface of the APIC. SSH, WebUI,
REST API all will use this address. Format uses IP/mask.
IPv4 gateway IP:
OOB interface default gateway.
Management speed/duplex: [auto]
Speed and duplex setting for OOB management interface. Valid values are as
follows: auto, 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full,
1000baseT/Full.
Enable strong passwords: [Y]
If “Yes” this will enforce password strength checking like on Nexus 7K. The
password must be at least 8 characters with one special character if enabled.
Admin password:
Once all of the information is gathered and vetted, the admin is ready to deploy
the initial APIC configuration.
Task 2: Initial ACI Fabric Configuration

In this task, students will be shown the Application Policy Infrastructure Controllers setup
process on an APIC Simulator. The simulator emulates 1 APIC, 2 Leafs, and 1 Spine in a
single appliance and gives the opportunity to demonstrate fabric initial configuration from
beginning to end.
The initial setup dialog is presented by the simulator, because this is a simulation, only a
single APIC is present and configured.
In a real ACI deployment, the initial setup is performed on all APIC appliances. Once the
devices have been powered on the setup is performed via the CIMC or console port on the
UCS chassis. The CIMC IP will need to be configured during the boot process using the
console or the UCS multi-port dongle. Once the CIMC address has been configured and
connected to the OOB network, it is available for SSH or HTTPs connections. If Serial-over-
LAN is enabled, an admin can SSH to the CIMC port and run “connect host” to drop into the
console. From the WebUI, the Java KVM console can be launched to gain access to the
console. The system will prompt for the initial configuration questions once the APIC has fully
booted.
NOTE: The screenshots here are from an APIC simulator. The actual APIC appliances do not
have a banner that says "STARTING APIC1".
Figure 1
Press ENTER to accept the default of "ACI Fabric1" for the fabric name Press ENTER to
accept the default of "1" for the fabric ID
Press ENTER to accept the default of "1" for the number of controllers Press ENTER to
accept the default of "1" for the POD ID
Press ENTER to accept the default of "1" for the controller ID
Press ENTER to accept the default of "apic1" for the controller name Press ENTER to
accept the default TEP address pool
Press ENTER to accept the default infra VLAN, 4 is used for the simulator Press ENTER to
accept the default multicast (GIPO) address pool
Press ENTER to accept the default of "N" for IPv6 OOB Mgmt
Figure 2
Enter the IP address for the OOB management interface of the APIC. This is the IP
address that will be used to access the APIC for the GUI and API calls
Enter the default gateway for the OOB network
Press ENTER to accept the interface speed/duplex mode to auto Enter the desired input for
strong password enforcement
Enter the desired password
Setup will prompt to confirm the accuracy of the information given
Figure 3
Review the configuration for accuracy

If there are no errors, hit the ENTER key
This will begin the ACI processes and complete the boot sequence
Figure 4
It takes several minutes for all of the processes to start on the APIC. The process that takes
the longest is authentication and it can take up to 5 minutes to start. During this time, attempts
to login will fail.
Once completed the admin will be able to login successfully and will be returned to an apic1#
prompt. At this point the admin should also be able to log into the APIC using the GUI.
Open a web browser (a shortcut to Chrome is available on the RDP desktop if following along
in the lab)
Navigate to the apic1 GUI using the OOB mgmt IP address configured in the initial
configuration: HTTPS://10.203.254.2XX
Note: The APIC will not automatically redirect from HTTP to HTTPs unless configured to do
so.
Select: ADVANCED to bypass the self-signed certificate warning
Figure 5
Click: Proceed to (IP address)
Figure 6
Login with the credentials supplied during the initial setup and ensure that the
ADVANCED MODE is selected. (The credentials should be "admin" "lumos123")
Note: If unable to login, return to the APIC console and verify the login credentials. As a
reminder, it takes several minutes for the APIC to fully boot, configure and start all
processes.
Figure 7
The first thing most will notice is a warning banner across the top of the screen indicating that
this is not a fully formed cluster since only one APIC is configured at this time. Once a full
cluster of APICs is established (at least three, but possibly five depending upon initial
configuration), this warning will disappear.
Note that in ACI 3.x and later this banner has been replaced by a warning icon in the top right of
the System Dashboard screen.
Note: In the lab environment students will only have 1 APIC assigned per fabric.
There is also a “What's New” pop-up that contains useful information and help on new
features.
Click: CLOSE on this pop-up window
Figure 8
If applicable, Click: NO on the warning message
Note: This setting informs a user if they are about to make changes to an object already in use.
It does not pop-up additional warnings for new items.
The System Dashboard is now displayed. The Dashboard provides a single-pane-of-glass

overview of the health of the fabric.
Figure 9
Select: CONTROLLERS
Expand: CONTROLLERS → APIC1 (Node 1)
Select: CLUSTER AS SEEN BY NODE
All the members of a cluster can be seen here along with their status. The current size of the
cluster is one because synchronization takes places across the fabric and the fabric has not
been discovered yet. Additional information may also be reviewed concerning the APICs
interfaces, the status of various components, etc.
Now that the APIC has been configured, the rest of the fabric will need to be discovered and
populated.
Figure 10
Select: FABRIC
The Fabric view is used to see and configure all of the items relevant to the physical fabric.
APICs, switches, fabric-wide policies all the way down to individual interface policies are
configured from this menu.
Select: INVENTORY in the sub-menu Select: FABRIC MEMBERSHIP
The ACI Fabric discovery process should begin discovering attached devices as soon as the
APIC is functional. Once these devices are discovered, they will show up in this menu with a
serial number, model number and role (if applicable).
On the right-hand pane, double-click on the row with the first serial number Provide the
Pod ID Provide the Node ID Provide the Node Name Select: UPDATE to save
Figure 11
15-30 seconds after clicking the update button a /32 IP address will be assigned to the leaf
switch with the serial number that was just updated. This IP address is called the INFRA IP and
is used for the VXLAN tunnel endpoint.
Figure 12
With the first fabric leaf switch registered, the APIC will automatically start discovering spine
switches in the fabric. These switches will also appear in the Fabric Membership view. In the
figure above, the fabric has found 4 additional switches, 2 leaf switches and a spine. The next
step will be repeated to register all the additional fabric devices.
Double-click on the next serial number
Provide the Pod ID Provide the Node ID Provide the Node Name
Select: UPDATE to save
Figure 13
Once the spines are discovered, it will take approximately 1 minute for the fabric to discover the
additional leaf switches. In larger environments, ACI will find all the leaf switches connected to
this spine.
A completed fabric discovery is shown below. Notice the supported model column. Some older
switches are not supported on newer code versions and vice-versa.
Figure 14
Now start examining the physical switches in the fabric.

In FABRIC → INVENTORY
Expand: POD 1 in the left-pane Select: LEAF3 (NODE-203)
Review the Dashboard for this specific switch (displayed on the right). In this switch specific
dashboard, all relevant health scores, faults and other details are accessible.
Figure 15
Select: INTERFACE in the right-hand pane
A port level view of Leaf 3 should be shown. This view is an easy way to check the status of a
port and if a known device is connected to the port.
Select: The GREEN PORT (48 in this figure) and the INTERFACE DETAILS should appear
with a drop-down that apic1 (controller) is connected to this port
Figure 16
Select: POD 1 from the left-hand pane

Select: TOPOLOGY from the right-hand pane
Figure 17
The topology view will now show the connections for the entire Pod 1 switch fabric including
APICs, Leaf switches and Spines.
Task 3: Fabric Pod and Access Policies

Once the fabric has been discovered and all of the nodes added, there are a few additional
components that need to be configured in order for the fabric as a whole to be able to pass
traffic. These “Pod Policies” will govern the management of the fabric itself. In this section,
students will be shown where the necessary Pod Polices are for management and
maintenance of the fabric, and how each is configured. As these steps can only be done
once per fabric, these items have already been completed in the lab environment and will not
be performed by students. This is meant only as a reference for situations outside the
classroom.
Activity Procedure
The first task in making sure that the fabric is operating efficiently is to ensure that the time is
correct and synced across all devices. NTP (Network Time Protocol) is critically important to the
fabric in managing the policy database, atomic counters, flow sequencing and certificate
management just to name a few items.
This step has already been completed in the lab. Follow the steps below to check the fabric
NTP configuration.
Navigate to: FABRIC → FABRIC POLICIES → POLICIES → POD → DATE AND TIME
Select: POLICY DEFAULT
Figure 18
The following steps are provided as a reference only. DO NOT COMPLETE THESE
STEPS IN THE LAB FABRIC!
To add additional NTP servers one only needs to click the + to the right of the NTP server box.
This brings up an additional pop-up window with more details.
Figure 19
Fill in the rest of the information and click the submit button to save the changes.
Figure 20
Another critically important component for the fabric is the BGP Route Reflector configuration.
For these Route Reflectors to distribute (reflect) routes inside the fabric using Multi-Protocol
BGP, the MP-BGP process must be running and the spine switches configured as BGP Route
Reflectors.
This step has already been completed in the lab. Follow the steps below to check the
configuration of the BGP Route Reflectors.
Navigate to: SYSTEM → SYSTEM SETTINGS
Select: BGP ROUTE REFLECTOR
Notice that Spine1 – (Node 101) has been configured as the RR for the fabric. If more spines
were present in this fabric, a minimum of 2 spines should be configured to provide redundancy.
Figure 21
There are several considerations to be made regarding the assignment of route reflectors
depending on the scale and design of the fabric. Multi-pod, multi-site and stretched fabric all
need to be carefully and purposefully designed with these scenarios in mind. For small to
medium sized fabrics (4 or less spines, no multipod), all spine nodes are generally configured
as route-reflectors.
Once a Pod Policy is created, it will need to be added to a Policy Group and applied to a Pod
Profile. This step has already been completed in the lab. Follow the steps below to verify that
the fabric has a Pod Policy Group with the appropriate NTP and BGP RR settings and that the
Policy Group is referenced by the Pod Profile.
Navigate to: FABRIC → FABRIC POLICIES → PODS → POLICY GROUPS
Select: PODPOL
Figure 22
Notice that all the policy drop-down boxes on the right-hand pane are empty, but that the
resolved policies all say default. In ACI if a policy is missing, the default policy will be applied, if
it can be found. To verify that the policy group is applied:
Navigate to: FABRIC → FABRIC POLICIES → PODS → PROFILES
Select: POD PROFILE DEFAULT
Figure 23
The final task that needs to be completed but students are unable to perform in the lab
scenario, is to define Virtual Port Channel (vPC Groups). vPC allows LACP active/active port
channeling to different physical devices. Each switch can only be part of one vPC pair. These
pairs are configured in the GUI after the discovery process. To verify the vPC group
configuration in the lab follow the steps below.
Navigate To: FABRIC → ACCESS POLICIES → POLICIES → SWITCH
Select: VIRTUAL PORT CHANNEL DEFAULT
Figure 24
The following steps to add an additional vPC pair are provided as a reference only. DO
NOT COMPLETE THESE STEPS IN THE LAB FABRIC.
Click: The + to the right of the Explicit VPC Protection Groups box
Provide: The vPC pair name, a unique vPC ID, the first switch of the pair and the second
switch of the pair
Click: SUBMIT to save the changes
Figure 25
Lumos recommends utilizing switch node numbers for vPC names. If the numbering is
consistent and all switches are in a vPC pair, then it should always be an odd and an even
switch in each pair. This allows for hitless upgrades for all dual connected hosts as the
recommended maintenance groups will be Odd and Even. This schema also makes
specifying the vPC Domain ID easy. Since the Node IDs and the vPC domain IDs must be
unique in each fabric, simply use the odd number node ID as the vPC domain ID and there
will never be any need to track separately or accidentally overlap domain IDs in a fabric. For
example, the vPC group of switches 211 and 212 would be named L211-212 and the vPC ID
would be 211.
Task 4: Out-of-Band (OOB) Management

One other critical task that is often missed during the initial fabric setup is that of connecting
the rest of the devices to the OOB network. Like the APICs, each fabric node has a MGMT0
interface that can be connected to the OOB network for management. This is not mandatory
however, and some admins prefer to manage the switches from the in-band (INB) network.
Lumos always recommends connecting devices to the OOB network if the option is available.
In the case of a failure in the in-band network, devices will still be reachable via the out-of-
band and this allows for management of switches (AAA, SNMP polling, trapping, etc.) on a
dedicated management network with separate security standards and dedicated links.
Activity Procedure
This task has already been completed in the lab environment. Follow the steps below to
verify that the OOB Management Configurations have been applied:
Navigate to: TENANTS → MGMT → NODE MANAGEMENT ADDRESSES
Expand: NODE MANAGEMENT ADDRESSES
Select: STATIC NODE MANAGEMENT ADDRESSES
Figure 26
Any nodes that have been configured with a static IP address will be shown in the right-hand
pane along with the assigned IP address, gateway and the type (OOB or INB).
The following steps are provided as a reference only. DO NOT COMPLETE THESE
STEPS IN THE LAB FABRIC!
In order to add additional ranges or nodes, the admin would perform the following actions:
Right-click: STATIC NODE MANAGEMENT ADDRESSES
Select: CREATE STATIC NODE MANAGEMENT ADDRESSES
Figure 27
In the Create Static Node Management Addresses pop-up window:

Name: This could be a single switch, i.e. Node Range: 101-101 Or multiple switches, i.e.
Node Range: 101-199
Provide the type of network (OOB, INB, or both)
Provide the desired network Management EPG
Provide IPv4 or IPv6 beginning address and mask.
If a range of nodes is provided, IPs will be assigned sequentially until the subnet/ mask is
exhausted
Provide the gateway address Click: SUBMIT
Figure 28
Now that an address policy has been completed it will need to be applied to a node
management address profile.
Click: DEFAULT directly under the Static Node Management Addresses folder on the left-
hand pane
Notice that the Node Blocks are applied to all of the nodes that are in the Static Node
Management Addresses policy that was seen in the last step. If new devices are to be added to
the policy, the node blocks must also be linked by clicking the + symbol to the right of the Node
Blocks box.
Figure 29
Name: Add the new Switch IDs in the field

Click: SUBMIT to finalize the changes and add the switch block to the profile
Once completed, this will assign IP addresses to the specified nodes on the network that has
been configured.
Figure 30
Task 5: APIC GUI Overview

In this task, students will examine and review the top-level options in the ACI GUI. The
System Dashboard, API Inspector, Vizore, L4-L7 Services, Virtual Networking, Admin,
Operations and finally, the Apps menu will be covered.
Activity Procedure
Click: The GEAR ICON on the far right of the screen as shown below
Familiarize yourself with all the options that can be done with this pull down. This includes API
Inspector, Debug Info and APIC About.
Figure 31
Click: L4-L7 SERVICES in the top menu as shown below

Notice there is both an INVENTORY and PACKAGES sub items as shown below. Read the
Quick Start help for this section.
Figure 32
Click: VIRTUAL NETWORKING in the top menu
Notice here that only an INVENTORY sub-menu exists. The INVENTORY menu displays the
VMs, hypervisors, and virtual switches belonging to the fabric. This menu also provides VM
statistics including packet counters, byte counters, CPU usage, and memory usage. Read the
Quick Start Help section.
Figure 33
Click: FABRIC in the top menu
You should already be familiar with the INVENTORY sub-menu
Figure 34
Familiarize yourself with the other two sub menu items.

Click: FABRIC POLICIES
Fabric Policies configure interfaces that connect spine and leaf switches and can be used to
enable features such as monitoring (statistics collection and statistics export), troubleshooting
(on-demand diagnostics and SPAN), or NTP (as shown in the previous tasks).
Figure 35
Click: ACCESS POLICIES
Access policies configure external-facing interfaces that do not connect to a spine switch.
External-facing interfaces connect to non-fabric such as virtual machine controllers and
hypervisors, hosts, routers, or fabric extenders (FEX). Access policies are used to configure
and enable items such as port channels and virtual port channels, protocols such as LLDP,
CDP or LACP, and features like monitoring or diagnostics.
Figure 36
Click: TENANTS
A Tenant is a logical container or a folder for application policies. This container can
represent an actual tenant, an organization, security zone, application or a domain.
Additionally, a Tenant can also just be used for the convenience of organizing information.
Tenants represent a unit of isolation from a policy perspective. Notice that there are three
Tenants preconfigured: common, infra and mgmt.
Figure 37
The common tenant is preconfigured for defining policies that provide a common behavior for
all the tenants in the fabric. A policy defined in the common tenant is usable by any other
tenant by default.
Click: SYSTEM from the top menu
Click: QUICK START on the sub-menu
These Quick Start menus are a very useful tool as you are learning about ACI and the APICs.
Quick Start sections will assist you in performing common and basic procedures, provide short-
cut wizards, reference material, help and concise instructional videos. Select the different icons
to the right of the topics listed to view an example of each.
Figure 38
Click: DASHBOARD on the sub-menu
As previously seen, the Dashboard provides a quick and concise overview of the system
health.
Figure 39
Click: CONTROLLERS on the sub-menu
As seen previously, the Controller displays property and status information about the APIC
instances and clusters.
Figure 40
At this point students should be comfortable navigating the top-level options of the APIC GUI.
This completes this exercise.
Fabric Access Policy Configuration
Table of Contents:
Task 1: Create a VLAN Pool
Task 2: Create a Physical Domain
Task 3: Create an Attachable Access Entity Profile
Task 4: Create the Interface Policies
Task 5: Create the Interface Policy Groups
Task 6: Create the Interface Profiles
Task 7: Create the Switch Profiles
Task 8: Confirm Interfaces to Switch Profile Association
Screenshots are provided to guide you through each step. These screenshots are based on
the lab using Tenant #1 as a visual aide. In many cases, you will need to replace the
information in these screenshots with your Student/Tenant # and/or information from a
particular reference table.
Activity Objective
Fabric access policies are the settings that control access to and from the fabric such as
VLANs, interface settings, vPC and port channel configurations and access port settings.
These settings form the basis for physical connectivity to the fabric.
In this activity, students will begin to configure the fabric access policies necessary for a
properly working fabric. Students will be creating Interface Policies, Policy Groups and
Profiles, Switch Policies and Profiles, Attachable Entity Profiles, VLAN Pools and Domains
and the relationships that tie these objects together. These constructs will be utilized as the
basis for the rest of the labs. When finished, students should have a basic understanding of
the components and workflow to configure the physical layer of the ACI fabric.
Each student will be executing these tasks for educational purposes, which will create
multiple identical copies of each policy under the Fabric → Access Policies section. In a true
production scenario, these steps would ideally need to be implemented once during the initial
setup and then only when additional port-groups need to be configured. To allow each
student a chance to complete this objective without object naming conflicts, we are using the
TXX naming convention in this shared configuration space.
Required Resources
• Access to the Lumos ACI fabric through the RDP server using information provided
below.
◦ Username: admin
APIC IP Address
https://10.203.254.24
RDP Connection
Tasks
Task 1: Create a VLAN Pool
The following figure provides an overview of the fabric access portion of the management
information tree (MIT) that we will be working with in this section. We will be following a
workflow that is exactly the same as one you would do in real life to establish Fabric Access
Policies. Our first step in the process is to create our VLAN pool.
Figure 41
In ACI, VLANs are allocated into pools. These pools can consist of one or more contiguous or
non- contiguous VLAN ranges. The pools will eventually determine which physical ports can
have a VLAN or VLANs associated with it by the relationships it has to other objects. These
relationships will be explained in greater detail as we go through the tasks in this lab.
In the lab, as in real world environments, our legacy network uses VLANs to isolate traffic.
Traffic ingressing or egressing the ACI fabric must retain these VLANs in order for devices
inside the fabric to communicate with those outside. One such device that is present in the lab,
as well as many real-world deployments, is the Cisco UCS Fabric Interconnect. The Fabric
Interconnects are not managed by ACI; however, they must be aware of the VLANs that will be
trunked into and out of the fabric.
For now, each student will set up a single VLAN pool for their entire tenant. This will contain the
VLANs that can be used dynamically for the VMM integration and the static VLANs required to
attach to the existing L2 and L3 Data Center network.
Activity Procedure
Follow the steps below to create a VLAN Pool to be used with your tenant throughout this
course.
Expand: POOLS
Right-click: VLAN
Select: CREATE VLAN POOL
Figure 42
Name: TXX-VLANPool
In production deployments, Lumos recommends using a single VLAN pool if possible. This
allows for all domains to be associated to a single VLAN pool, simplifying configurations. If
multiple VLAN pools are required, Lumos recommends simple naming schema that identifies the
purpose of the VLAN pool (i.e. Phys, L3Out, etc.), rather than the VLANs that are in the pool as
these can change.
Allocation Mode: DYNAMIC ALLOCATION

Left-click + under the Encap Blocks section to create an entry
Figure 43
Enter your Tenant's VLAN range (see table)
VLAN Ranges
Tenant Dynamic Range Static Range
T01 2010-2014 2015-2019
T02 2020-2024 2025-2029
T03 2030-2034 2035-2039
T04 2040-2044 2045-2049
T05 2050-2054 2055-2059
T06 2060-2064 2065-2069
T07 2070-2074 2075-2079
T08 2080-2084 2085-2089
T09 2090-2094 2095-2099
T10 2100-2104 2105-2109
T11 2110-2114 2115-2119
T12 2120-2124 2125-2129
T13 2130-2134 2135-2139
T14 2140-2144 2145-2149
T15 2150-2154 2155-2159
T16 2160-2164 2165-2169
Allocation Mode: DYNAMIC ALLOCATION (Inherit allocMode from parent is the default
setting)
Enter in the VLAN Range: 2XX0 to 2XX4
Click: OK to save
Figure 44
Add the Static VLAN allocation by selecting the + again under the Encap Block Section
Figure 45
Add in the VLAN Range: 2XX5 to 2XX9

Select the option for STATIC ALLOCATION for this VLAN Range
Click OK to save
Figure 46
Select SUBMIT to save the VLAN Pool

Your VLAN Pool configuration should now be like what is displayed below
Figure 47
Task 2: Create a Physical Domain

In ACI, all End Point Groups require a domain association. Domains act as the glue between
the configuration done in the fabric tab to the policy model and endpoint group configuration
found in the tenant tab. The fabric administrator creates the domains, and the tenant
administrators associate domains to endpoint groups.
Physical Domains associate VLAN Pools and Attachable Access Entity Profiles and are tied
to physical ports on the ACI leaf switches through the EPG. External Routed Domains are
used to associate VLAN pools for L3Outs. External Bridged Domains do the same for
L2Outs. The APIC checks if an EPG is associated with one or more of these types of
domains. If the EPG is not associated, the system accepts the configuration but raises a fault.
The deployed configuration may not function properly if the domain association is not valid.
For example, if a user configures a VLAN for an EPG that is not part of a pool that the EPG is
associated with (via the Domains) the system will raise a fault, and traffic may not flow
properly.
Figure 48
Activity Procedure
Follow the steps below to create a Physical Domain to be used with your tenant.
Expand: PHYSICAL AND EXTERNAL DOMAINS
Right-click: PHYSICAL DOMAINS
Select: CREATE PHYSICAL DOMAIN
Figure 49
Name: TXX-Physical
Click: SUBMIT to save and close the window
Figure 50
Task 3: Create an Attachable Access Entity Profile

An Attachable Entity Profile (AEP) represents a group of external entities with similar
infrastructure policy requirements. The infrastructure policies consist of physical interface
policies that configure various protocol options, such as Cisco Discovery Protocol (CDP), Link
Layer Discovery Protocol (LLDP), or Link Aggregation Control Protocol (LACP).
An AEP is required to deploy VLAN pools on leaf switches. Encapsulation blocks (and
associated VLANs) are reusable across leaf switches. An AEP essentially ties the VLAN
Pools/Domains to the physical infrastructure. You can think of this sort of like a "switchport
trunk allowed VLAN" command, except we aren't trunking any VLANs, just allowing for them
to be used on a port.
The AEP defines the range of allowed VLANS, but it does not provision them. No traffic flows
unless an EPG is deployed on the port. Without defining a VLAN pool in an AEP, a VLAN is
not enabled on the leaf port even if an EPG is provisioned. Attached entity profiles can also
be associated directly with application EPGs, which deploy the associated application EPGs
to all those ports associated with the attached entity profile.
An Attachable Access Entity Profile (AEP) will be used later to tie tenant-specific
configuration to physical port configurations on the fabric.
Students will now need to create a single AEP for the VLAN pool and domain association.
In real world deployments, we recommend utilizing a single AEP for the entire fabric. The
exceptions to that recommendation are for multiple VMM domains (each requires its own
AEP) or for overlapping VLAN usage between tenants (each would require its own AEP).
Lumos also recommends following the same approach to naming of the AEPs as other
objects -- specifying the function (L3Out, Phys, VMM, etc.) as part of the name simplifies
identification and configuration.
Figure 51
Activity Procedure
Follow the steps below to create an AEP to be used for your tenant.
Expand: POLICIES → GLOBAL
Right-click: ATTACHABLE ACCESS ENTITY PROFILE
Select: CREATE ATTACHABLE ACCESS ENTITY PROFILE
Figure 52
Name: TXX-AEP
Select: NEXT
Figure 53
Do not make any changes to the next screen
This display may have more lines than you see below. It will depend on what tasks other
students have completed
Click: FINISH to save
Figure 54
Task 4: Create the Interface Policies

In ACI, interface policies define settings that are applied to physical interfaces (ports) on the
leaf switches. Policies can be created once and reused again many times. These policies are
for settings like CDP, LLDP, Port-Channel and vPC settings, link speed and duplex, storm
control, etc.
Figure 55
Activity Procedure
In this task, students will create all of the various interface policies needed for their tenants to
be functional. Normally, this would only require 1 set of policies per setting. For example,
admins would normally create a single CDP-ENABLE and CDP-DISABLE policy and then re-
use this policy as needed. For the purposes of the lab however, each student will create their
own policies to be used.
For the lab interface characteristics, we will create settings for CDP, LLDP and for access
ports. These settings can and will be used in multiple Policy Groups. Students will also create
uniquely named Policies for their Tenant.
In production deployments, we recommend that the interface policy naming convention

simply be consistent and done in a manner that makes sense to your team. For example,
using CDP-Enabled or CDP-Disabled (or even CDP-On and CDP-Off) very clearly identifies
the function and the setting in the name making it easy to figure out as admins are later
adding these settings to policy groups.
To create the CDP Interface Policies for your tenant:

Expand: POLICIES → INTERFACE
Right-click: CDP INTERFACE
Select: CREATE CDP INTERFACE POLICY
Figure 56
Name: TXX-CDP-Enabled
Admin State: ENABLED
Click: SUBMIT to save
Figure 57
Repeat the above process to create a CDP Disabled policy.

Name: TXX-CDP-Disabled
Admin State: DISABLED
Figure 58
To create the LLDP Policies:

Right-click: LLDP INTERFACE
Select: CREATE LLDP INTERFACE POLICY
Figure 59
Name: TXX-LLDP-Enabled
Receive State: ENABLED
Transmit State: ENABLED
Figure 60
Repeat the above process to create a LLDP Disabled policy.
Name: TXX-LLDP-Disabled
Receive State: DISABLED
Transmit State: DISABLED
Figure 61
Create MAC-Pinning Policies:

Right-click: PORT CHANNEL
Select: CREATE PORT CHANNEL POLICY
Figure 62
Name: TXX-PC-MacPinning
Select Mode: MAC PINNING
Figure 63
Task 5: Create the Interface Policy Groups

Interface policy groups are exactly what they sound like, a group of individual policies. They
are templates (like port profiles on Nexus 7k/5k) that dictate all aspects of port behavior and
are associated to an AEP. Interface policy groups use the policies configured in the previous
task to specify how individual ports should behave. There are three types of interface policy
groups depending on the link type: Access Port, Port Channel, and vPC.
For Port Channels and vPCs, each policy group designates a single logical interface on the
switches. Essentially, the port channel or vPC is equivalent to adding a channel-group XX
command to a Nexus 7K switch port. If the same XX number is used, the switch tries to
configure all ports as part of the same logical port-channel interface. If it is desired to create
10 PCs/vPCs then 10 separate policy groups must be created. However, access port policy
groups can be reused between interfaces. Policy groups do not actually specify where the
protocols and port behavior should be implemented. The "where" happens by associating
one or more interface profiles to a switch profile, covered in the following tasks.
In this task, students will create the Access Interface Policy Group that you will use for your
assigned ESXi server. This Access Policy Group will reference the interface policies created
in the previous tasks. Later, students will create a vSwitch policy configuration for the blade
servers themselves.
Naming conventions for port groups vary widely based on customer preference. Lumos
suggests a naming convention that either identifies the port in the name or the hostname of
the device it is attached to depending on team preference.
Figure 64
Activity Procedure

Expand: INTERFACES → LEAF INTERFACES → POLICY GROUPS
Right-click: LEAF ACCESS PORT
Select: CREATE LEAF ACCESS PORT POLICY GROUP
Figure 65
Name: TXX-FIA-PG
CDP Policy: TXX-CDP-Disabled
LLDP Policy: TXX-LLDP-Enabled
Attached Entity Profile: TXX-AEP
Note that for any policy fields that are not selected (for example, Link in the above task) the
system defaults are automatically deployed, however it is not shown in the output
Figure 66
Repeat the process for FI-B:

Right-click: LEAF ACCESS PORT
Select: CREATE LEAF ACCESS PORT POLICY GROUP
Figure 67
Name: TXX-FIB-PG
Figure 68
Task 6: Create the Interface Profiles

Interface profiles help tie all the different policy pieces together. Interface profiles contain
blocks of ports, called interface selectors, and are also associated to the interface policy
groups configured in the previous tasks. The Interface Profile by itself is just an arbitrary
container with a name, such as e1, or Leaf1. As such, the Interface Profile must be
associated to a specific Switch Profile to configure the ports to be ready for use. This
ultimately associates the Interface Selectors (which select ports) to a Switch Profile/Selector -
- deploying a configuration for X ports (as identified by the Interface Profile and it's child
object the Interface Selectors) to X switches (as identified by the Switch Profile and it's child
object, the Switch Selectors).
In this task, we will create the Interface Policies needed for each tenant’s physical port
connections. This Interface Policy Profile contains the ACI fabric leaf switches and interfaces
specific to the ESXi servers used in coming tasks. Refer to the table for your student/tenant-
specific UCS fabric interfaces.
Lumos recommends that naming conventions for interface profiles follow the same guidelines
we have set out in other places, keep it simple enough to quickly and accurately convey the
purpose of the object. In this task, students will use TXX-L1-IntProf. But a more real-world
example would be L101, meaning these are the interface selectors for Leaf Node 101.
If vPC will be used, a single interface profile can be created for both switches (L101-102).
When the e1/1 interface selector is applied to this profile, it will be active on switches 101 and
102 e1/1. The idea here is to create once and re-use as often as possible to reduce clutter
and make administration of the fabric easier.
For a complete vPC pair you might end up with L101, L101-102 and L102 interface profiles.
For the interface port selectors, a simple p1 for “port 1” or e1_1 for “eth1/1” is enough to
convey the usage and the meaning without being overly complex.
Figure 69
Activity Procedure

Expand: INTERFACES → LEAF INTERFACES
Right-click: PROFILES
Select: CREATE LEAF INTERFACE PROFILE
Figure 70
Name: TXX-L1-IntProf
Click: + to add an Ethernet Interface to create an entry
Figure 71
Access Port Selector Name: TXX-FIA-Port<#>

Example: T01-FIA-Port01
Interface: 1/ (See table below for your tenant's specific interface)
Fabric Access Information

Tenant Policy Group Type ACI Leaf 201 FI-A Port
T01 Access Port 1/1 1/1
Interface Policy Group Pull-Down: TXX-FIA-PG

Click: OK to save
Figure 72
Figure 73
Repeat the process for FI-B.

Figure 74

Name: TXX-L2-IntProf
Click: + to add an Ethernet Interface to create an entry
Figure 75
Access Port Selector Name: TXX-FIB-Port<#>

Example: T01-FIB-Port01
Interface: 1/ (See table below for your tenant's specific interface)
Fabric Access Information
Tenant Policy Group Type ACI Leaf 202 FI-B Port
Interface Policy Group Pull-Down: TXX-FIB-PG

Click: OK to save
Figure 76
Figure 77
Task 7: Create the Switch Profiles

Switch Profiles allow for the selection of one or more leaf switches. The Switch Profiles are
then tied to Interface Profiles to apply the desired configuration to the ports on that specific
switch node. This association pushes the configuration to the interface and creates the Port
Channel or vPC (if one has been configured) in the interface policy.
For this task, students will create a Switch Profile for each individual switch. In another lab
exercise, students will create a Switch Profile for a vPC pair for devices that are dual
connected.
Lumos recommends that the naming conventions of Switch Profiles mirror exactly that of your
Interface Profiles in a production deployment. This eliminates any confusion about how these
items should be related.
Figure 78
Activity Procedure

Expand: SWITCHES
Expand: LEAF SWITCHES
Select: CREATE LEAF PROFILE
Figure 79
Name: TXX-L1-SP
Left-Click + next to Leaf Selectors
Leaf Selector Name: TXX-L1-SS
Switch (Blocks): 201 (leaf1)
Select: UPDATE
Figure 80
Click: NEXT
Figure 81
Find the Interface Profile you created for leaf1 in the last task (TXX-L1-IntProf) and
select the check-box next to it.
Click: FINISH
Figure 82
Repeat the process for Leaf 2.
Expand: SWITCHES
Expand: LEAF SWITCHES
Name: TXX-L2-SP
Click: + next to Leaf Selectors
Leaf Selector Name: TXX-L2-SS
Switch (Blocks): 202 (leaf2)
Select: UPDATE
Figure 83
Click: NEXT
Figure 84
Find the Interface Profile you created for leaf2 in the last task (TXX-L2-IntProf) and
select the check-box next to it
Click: FINISH
Figure 85
Task 8: Confirm Interfaces to Switch Profile Association
Now that the Interface profiles have been completed, the association to the switch profile
containing leaf1 and leaf2 needs to be configured.
Activity Procedure
Navigate to: FABRIC → ACCESS POLICIES → SWITCHES → LEAF SWITCHES →

PROFILES → TXX-L1-SP
Your results should be comparable to that shown below.
Figure 86
In most cases, when looking at relationships between objects, the ACI GUI will indicate if there
is a problem by using the state field. If the state is “formed”, that means that the object exists in
the MIT and is of the right class type. It does not however, mean that the settings of that object
are correct. If the relationship reports “missing” this is an indication that a step was skipped, or
an object is not named exactly the same as the reference object. Remember that case matters.
Tenant Application Profile Configuration
Table of Contents:
Task 1: Create a Tenant
Task 2: Create a VRF for your Tenant
Task 3: Create your Tenant's Bridge Domains
Task 4: Create Subnets for Each Bridge Domain
Task 5: Create the Application Network Profile
Task 6: Create the End Point Groups
reference table.
Activity Objective
In this activity, students will begin to configure the tenant policies necessary for a properly
working fabric. Students will be creating Tenants, VRFs, Bridge Domains and assigning
subnets, Application Network Profiles, End-Point Groups and the relationships that tie these
objects together to add to the fabric access constructs created in the previous lab. These
constructs will be utilized as the basis for the rest of the labs. When finished, students should
have a basic understanding of the components and workflow to configure the logical layer of
the ACI fabric.
Required Resources

• Access to the Lumos ACI fabric through the RDP server using information
provided below.
◦ Username: admin
• APIC IP Address
https://10.203.254.24
RDP Connection
Tasks
Task 1: Create a Tenant
Tenants are the top-level containers for application policies. They are also logical containers
that enable administrators to exercise domain-based access control. A tenant represents a
unit of isolation from a policy perspective, but it does not represent a private network. Tenants
can represent a customer in a service provider setting, an organization or domain in an
enterprise setting, or just a convenient grouping of policies. The following figure provides an
overview of the tenant portion of the Management Information Tree (MIT) we will be working
with in our labs.
Figure 87
There is no best practice recommendation for tenant layout. It will depend heavily on scale,
desired traffic flow and security requirements inside the fabric.
Each student will create their own tenant to use as the basis for the rest of the labs.
Activity Procedure
Navigate to: TENANTS

Click: ADD TENANT
Figure 88
Name: TXX
Click: SUBMIT
Figure 89
Although you can configure much of the initial configuration using the wizards and/or drag
and drop interfaces, we will add the Tenant networking manually. This allows us better
inspection of how all these constructs tie in and work together.
Task 2: Create a VRF for your Tenant

A Virtual Routing and Forwarding (VRF) or context is a tenant network (also referred to as
private network in the older APIC versions). A tenant can have multiple VRFs. A VRF is a
unique Layer 3 forwarding and application policy domain. At the most basic, VRFs provide
isolated IP routing tables just as they do in legacy environments. This allows for overlapping
IP space to be configured in different VRFs without conflicts.
In this task, students will add a VRF to their tenant which will provide the foundation for the
networking constructs.
Figure 90
Activity Procedure
Navigate to: TENANTS → TXX → NETWORKING

Right-click: VRFS
Select: CREATE VRF
Figure 91
Name: MAIN
Remove the check from the box labeled CREATE A BRIDGE DOMAIN
Figure 92
Task 3: Create your Tenant's Bridge Domains

Bridge Domains represent Layer 2 forwarding constructs within the fabric. The bridge domain
defines the unique Layer 2 MAC address space and a Layer 2 flood domain if such flooding
is enabled. While a VRF defines a unique IP address space, that address space can consist
of multiple subnets. Those subnets are defined in one or more bridge domains that reference
the corresponding VRF. Each bridge domain must be tied to a VRF.
In this task, students will create three bridge domains; one for web, database, and ERSPAN
functions.
Figure 93
Activity Procedure

Right-click: BRIDGE DOMAIN
Select: CREATE BRIDGE DOMAIN
Figure 94
Name: Web
Type: REGULAR
VRF: TXX/Main
Forwarding: OPTIMIZE
Click: NEXT
Figure 95
Click: NEXT on the L3 Configuration section, no configuration is necessary

currently
Figure 96
Click: FINISH
Figure 97
Your Bridge Domain should look like that shown below.
Figure 98
Create a second Bridge Domain for your DB tier.

Right-click: BRIDGE DOMAIN
Select: CREATE BRIDGE DOMAIN
Figure 99
Name: DB
Type: REGULAR
VRF: TXX/Main
Click: NEXT
Figure 100
Click: NEXT
Figure 101
Click: FINISH
Figure 102
Task 4: Create Subnets for Each Bridge Domain

A bridge domain can contain multiple subnets, but a subnet is contained within a single
bridge domain. If the bridge domain Limit IP Learning to Subnet property is set to 'yes',
endpoint IP learning will occur in the bridge domain, if the 'limit learning to subnet' box is
checked, IPs will be learned only if the IP address is within any of the configured subnets for
the bridge domain or within an EPG subnet when the EPG is a shared service provider.
Subnets can span multiple EPGs; one or more EPGs can be associated with one bridge
domain or subnet.
Students will create the IP subnets used for the previously created Bridge Domains. This will
enable the gateway SVI for the VM Guests we will be deploying in later labs.
Figure 103
Activity Procedure
Navigate to: TENANTS → TXX → NETWORKING → BRIDGE DOMAIN → Web

Right-click: SUBNETS
Select: CREATE SUBNET
Figure 104
Gateway IP: (See Table)
Lab 3 - Table 1
Tenant Web DB SPAN
T01 10.1.1.1/24 10.1.2.1/24 10.1.3.1/24
T02 10.2.1.1/24 10.2.2.1/24 10.2.3.1/24
T03 10.3.1.1/24 10.3.2.1/24 10.3.3.1/24
T04 10.4.1.1/24 10.4.2.1/24 10.4.3.1/24
T05 10.5.1.1/24 10.5.2.1/24 10.5.3.1/24
T06 10.6.1.1/24 10.6.2.1/24 10.6.3.1/24
T07 10.7.1.1/24 10.7.2.1/24 10.7.3.1/24
T08 10.8.1.1/24 10.8.2.1/24 10.8.3.1/24
T09 10.9.1.1/24 10.9.2.1/24 10.9.3.1/24
T10 10.10.1.1/24 10.10.2.1/24 10.10.3.1/24
T11 10.11.1.1/24 10.11.2.1/24 10.11.3.1/24
T12 10.12.1.1/24 10.12.2.1/24 10.12.3.1/24
T13 10.13.1.1/24 10.13.2.1/24 10.13.3.1/24
T14 10.14.1.1/24 10.14.2.1/24 10.14.3.1/24
T15 10.15.1.1/24 10.15.2.1/24 10.15.3.1/24
T16 10.16.1.1/24 10.16.2.1/24 10.16.3.1/24
Add Gateway IP: See table for IP address

Click: Submit
Figure 105
Create another subnet in the DB Bridge Domain

Navigate to: TENANTS → TXX → NETWORKING → BRIDGE DOMAIN → DB
Right-click: SUBNETS
Select: CREATE SUBNET
Figure 106
Gateway IP: (See Table)
Lab 3 - Table 1
Tenant Web DB SPAN
T01 10.1.1.1/24 10.1.2.1/24 10.1.3.1/24
T02 10.2.1.1/24 10.2.2.1/24 10.2.3.1/24
T03 10.3.1.1/24 10.3.2.1/24 10.3.3.1/24
T04 10.4.1.1/24 10.4.2.1/24 10.4.3.1/24
T05 10.5.1.1/24 10.5.2.1/24 10.5.3.1/24
T06 10.6.1.1/24 10.6.2.1/24 10.6.3.1/24
T07 10.7.1.1/24 10.7.2.1/24 10.7.3.1/24
T08 10.8.1.1/24 10.8.2.1/24 10.8.3.1/24
T09 10.9.1.1/24 10.9.2.1/24 10.9.3.1/24
T10 10.10.1.1/24 10.10.2.1/24 10.10.3.1/24
T11 10.11.1.1/24 10.11.2.1/24 10.11.3.1/24
T12 10.12.1.1/24 10.12.2.1/24 10.12.3.1/24
T13 10.13.1.1/24 10.13.2.1/24 10.13.3.1/24
T14 10.14.1.1/24 10.14.2.1/24 10.14.3.1/24
T15 10.15.1.1/24 10.15.2.1/24 10.15.3.1/24
T16 10.16.1.1/24 10.16.2.1/24 10.16.3.1/24
Add Gateway IP: See table for IP address

Click: Submit
Figure 107
Task 5: Create the Application Network Profile

An application profile defines the policies, services and relationships between endpoint
groups (EPGs). Application profiles contain one or more EPGs as modern applications
contain multiple components to provide a service. For example, an e-commerce application
could require a web server, a database server, data located in a storage area network, and
access to outside resources that enable financial transactions. The application profile
contains as many (or as few) EPGs as necessary that are logically related to providing the
capabilities of an application.
In ACI deployments Application Profiles are often used simply as a "folder" for organizational
purposes rather than their intended purpose of identifying a application or set of applications.
Many deployments have a "DMZ" Application Profile, or a "Prod" Application Profile with all
EPGs for that tier housed within. This is a very simple way to organize the fabric, however it
has the drawback of providing less detailed health reports on specific applications. For
example, if multiple applications reside in the "Prod" Application Profile, and one is having
issues, the overall health score of the "Prod" Application Profile will decrement, however not
nearly as much as if that application was contained in a dedicated Application Profile.
In this task, students will create an application profile for the lab application.
Figure 108
Activity Procedure
Navigate to: TENANTS → TXX

Right-click: APPLICATION PROFILES
Select: CREATE APPLICATION PROFILE
Figure 109
Name: WebApp
Figure 110
Task 6: Create the End Point Groups

End Point Groups are perhaps the most important object in the policy model. An EPG is a
logical object that contains a collection of endpoints. EPGs are fully decoupled from the
physical and logical topology.
Endpoints are devices that are connected to the network directly or indirectly. They have an
address, location, attributes, and can be physical or virtual. Endpoint examples include
servers, virtual machines, network-attached storage, or clients on the Internet. Endpoint
membership in an EPG can be dynamic or static.
Policies apply to and are enforced at EPGs, not to individual endpoints (with some exceptions
for micro- segment EPGs). An EPG can be statically configured by an administrator in the
GUI, or dynamically configured by an automated system such as VMM.
In this task, students will create a Web and an DB EPG for our test application. These EPGs
will be where we create and apply policy to the individual endpoints (hosts) attached to the
fabric.
Figure 111
Activity Procedure
Navigate to: TENANTS → TXX → APPLICATION PROFILES → WebApp

Expand: WebApp
Right-click: APPLICATION EPGS
Select: CREATE APPLICATION EPG
Figure 112
Name: Web
Bridge Domain: TXX/Web
Figure 113
Repeat the same for the DB EPG.

Navigate to: TENANTS → TXX → APPLICATION PROFILES → WebApp
Expand: WebApp
Right-click: APPLICATION EPGS
Select: CREATE APPLICATION EPG
Figure 114
Name: DB
Bridge Domain: TXX/DB
Click: FINISH to save.
Figure 115
This now fulfills the basic requirements for a working fabric. All we need to add now are
endpoints, which will be done in the next lab.
VMM Integration
Table of Contents:
Task 1: Create a VMM Domain
Task 2: Create vSwitch Policies
Task 3: Verify VMM Domain Integration
Task 4: Attach the ESXi Servers to the Virtual Distributed Switch
Task 5: VMM to EPG Associations
Task 6: Assign Virtual Machines to vDS Portgroups
Task 7: Verify VM Connectivity
Activity Objective
By now, students should have a good understanding of what benefits VMM integration
entails, and the basics of how to configure it. In this activity, students will configure Virtual
Machine Manager (VMM) for integration between VMWare vSphere and Cisco ACI. Students
will be creating a vSphere controller, supplying vCenter credentials and pushing port groups
down to vCenter from ACI. The End Points (EPs) students will use for testing reside on virtual
guests hosted on these vCenters, successful completion of this lab validates all steps taken
so far. When finished, students should have a basic understanding of the components, the
workflow to configure the VMM integration with the ACI fabric, how to assign a port group to a
VNIC from vSphere, and how to log into the Virtual Guests.
Required Resources

provided below.
◦ Username: admin
• APIC IP Address
https://10.203.254.24
RDP Connection
Tasks
Task 1: Create a VMM Domain
VMM domains contain VM controllers such as VMware vCenter or Microsoft SCVMM and the
credential(s) required for the ACI API to interact with the VM controller. A VMM domain
allows for VM mobility within the domain but not across domains -- meaning that ACI does
not provide any cross VMM domain migration capabilities. A single VMM domain can contain
multiple instances of a hypervisor within a VMM controller but they must be the same kind.
For example, a VMM domain can contain many ESXi instances running multiple VMs but it
may not also contain Hyper-V hosts. A VMM domain inventories controller elements (such as
pNICs, vNICs, VM names, and so forth) and pushes policies into the controller(s), creating
port groups, and other necessary elements. The ACI VMM domain listens for controller
events such as VM mobility and responds accordingly.
Figure 116
Activity Procedure
Navigate to: VIRTUAL NETWORKING → INVENTORY

Expand: VMM DOMAINS
Right Click: VMWARE
Select: Create VCENTER DOMAIN
Figure 117
Name: TXX-vCenter
Virtual Switch: VMWARE VSPHERE DISTRIBUTED SWITCH (default)
Associated Attachable Entity Profile: TXX-AEP
VLAN Pool: TXX-VLANPool (dynamic)
vCenter Credentials: Click + to add an entry
Figure 118
Name: TXX-Credentials (See Table)
vCenter Information
Tenant vCenter Login Password Datacenter
T01 10.203.254.31 administrator@vsphere.local lumos123 Tenant01

Click: OK to save
Figure 119
vCenter: Click + to add an entry

Figure 120
Name: TXX-vCenter
Hostname (or IP Address): IP OF YOUR TENANT'S VCENTER (see table)
DVS Version: DVS VERSION 6.0
Datacenter: TenantXX (case sensitive!)
Associated Credential: TXX-Credentials
Click: OK to save
vCenter Information

Figure 121

Figure 122
If there were no mistakes in the configuration, you'll see the vCenter information in the APIC
(as shown below). If this is blank, you probably have an error in your configuration and you
may need to investigate the faults and/or delete and recreate the individual objects.

Figure 123
Task 2: Create vSwitch Policies

By default, on the Virtual Distributed Switch, the Discovery Protocol used is LLDP. This is fine
for any servers that support LLDP, but some servers do not support LLDP unless certain code
revisions, driver firmware patches, etc. are used. Because of this, ESXi cannot report LLDP
information to the APIC, unless you are using these certain code levels or hardware. As an
alternative to LLDP, use CDP to discover the hosts. In order to set the DVS to use CDP, you
would need to configure a vSwitch policy on the Attachable Entity Profile (AEP) that has CDP
and LLDP enabled.
In the Lumos lab environment the student ESX instances are "nested ESX" instances -- this
means that the vSwitch deployed to each student instance will not "see" the Fabric
Interconnects via CDP or LLDP as their "next hop" is the parent ESX instances they reside in.
This causes an issue for ACI as it can no longer locate the placement of Virtual Machines via
CDP/LLDP, because ACI sees the Fabric Interconnects, however the nested vSwitch does not
see the Fabric Interconnects. Because of this nested ESX environment, the use of Pre-

Provision forces ACI to deploy the configurations required for VM communication even
though it has not learnt the location of the virtual machines. This is NOT a "normal" real-world
scenario -- in production, Pre- Provision should not be used unless necessary.
Students will now configure the vSwitch policy with the appropriate settings for the ESXI
hosts attached to each tenant.
Activity Procedure
Navigate to: VIRTUAL NETWORKING → INVENTORY → VMM Domains → VMWARE →

TXX- VCENTER
Under vSwitch Policies select:

Port Channel Policy: TXX-PC-MACPinning
LLDP Policy: TXX-LLDP-Disabled
CDP Policy: TXX-CDP-Enabled
Click: SUBMIT to save (If the SUBMIT button does not allow you to select, don't worry, this
is not a critical step in this lab environment)
Figure 124
Task 3: Verify VMM Domain Integration

To verify that your vCenter Integration is successful, you can check in several places.
Activity Procedure
Navigate to: VIRTUAL NETWORKING → INVENTORY → VMM Domains → VMWARE →

TXX- VCENTER
Expand: CONTROLLERS → TXX-VCENTER → HYPERVISORS
If your tenant vCenter shows servers attached under your hypervisor folder, integration has been
successful

Figure 125
Another place to verify integration has been completed is from the vCenter itself. From your
terminal server's desktop, you can open the vSphere client and login to your vCenter server
and follow the screen shot to see that the APIC has created a vDS inside of the vCenter
Server.
Verification from vSphere Procedure
On the RDP Server Desktop, double-click the VMware vSphere icon (pictured below)
Figure 126
IP Address / Name: IP ADDRESS OF YOUR TENANT'S VCENTER (See Table)
vCenter Information

Figure 127
A Certificate Warning message may appear; Click: IGNORE

Figure 128
Once logged in, you can move to the Networking view to see if the APIC vDS has been
successfully attached to the vCenter Server.
Navigate to: HOME → INVENTORY → NETWORKING
Expand: aci03-TXX-VCENTER → TENANTXX → TXX-VCENTER
The presence of the TXX-vCenter object further validates that your VMM integration was
successful. Your screen should display output like that shown below.
Figure 129
Task 4: Attach the ESXi Servers to the Virtual Distributed Switch

We have now associated the VMM in ACI and created the vDS in vCenter. This next task will
have students attach their ESXI hosts to the vDS in order to allow vNICs to be allocated to
guests.
Activity Procedure
Navigate to: HOME → INVENTORY → NETWORKING → TXX-VCENTER

Expand: TENANTXX → TXX-VCENTER (folder)
Select: TXX-VCENTER Virtual Distributed Switch in the left pane
Click: ADD HOST... from the Summary tab of the right pane, or via the right-click context
menu

Figure 130
In the next screen, vmnic1 of Host 1 and Host 2 should not be in use by any switch. This is
denoted by a "---" in the second column of the right pane.
NOTE: Hosts may not show up in sequential order -- look for the "---" indication to ensure you
are selecting the correct hosts! You want to only select the 2 lowest numbered IP hosts for your
tenant. Do not select all 3 hosts or you will have trouble in later labs.
Select: CHECK BOX next to vmnic1 of Host 1
Select: CHECK BOX next to vmnic1 of Host 2
DO NOT select any other hosts or vmnics!
Click: NEXT

Figure 131
No configuration is required on the next screen Click: NEXT
Figure 132

No configuration is required on the next screen Click: NEXT
Figure 133
No configuration is required on the next screen Click: FINISH

Figure 134
You should now see two hosts connected to the vCenter DVS.
Figure 135
Task 5: VMM to EPG Associations

In ACI, each End Point Group (EPG) needs to be tied to a domain, physical or virtual. This is
the glue that ties together our fabric access policies we created in Lab 2 and the tenant
policies we created in Lab 3.
Now that the server-side configuration of the virtual Distributed Switch (vDS) is complete, we
are now ready to begin tying the individual EPGs we created in ACI to VMWare vDS Port
Groups that will be created by completing the next task.

Figure 136
Activity Procedure
Navigate to: TENANT → TXX → APPLICATION PROFILES → WEBAPP →

APPLICATION EPGS → WEB
Expand: Web
Right Click: DOMAINS (VMS AND BARE-METALS)
Select: ADD VMM DOMAIN ASSOCIATION

Figure 137
VMM Domain Profile: VMWARE/TXX-VCENTER

Deploy Immediacy: IMMEDIATE
Resolution Immediacy: PRE-PROVISION (This is Critical!)
VLAN Mode: DYNAMIC
Click: SUBMIT

Figure 138
Using your vCenter client, you can check and see that a new port-group should now be
created within the vDS tied to your EPG, using a TENANT|ANP|EPG naming convention.

Figure 139
You can see the VLAN that was assigned from the Dynamic Pool.
Select: TXX|WebApp|Web
Select: MANAGE THIS DISTRIBUTED PORT GROUP
Another window will appear, select: VLAN and verify the VLAN ID
Figure 140

Perform the same steps for EPG DB under the Application Profile WebApp.
Navigate to: TENANT → TXX → APPLICATION PROFILES → WEBAPP →

APPLICATION EPGS → DB
Expand: DB
Right Click: DOMAINS (VMS AND BARE-METALS)
Figure 141

Resolution Immediacy: PRE-PROVISION (This is Critical!)
VLAN Mode: DYNAMIC
Click: SUBMIT

Figure 142
After clicking SUBMIT, a second newly-created vDS port-group should appear for DB
within vSphere as well.
Task 6: Assign Virtual Machines to vDS Portgroups

In this task, students will assign the vNICs to the VMs. For each Virtual Machine, we will need to
go modify the network adapter port-group to the appropriate EPG port-group. This step will need
to be performed for all VMs that require communication through ACI. This configuration is not
performed automatically. Note that by keeping our Tenant, ANP, and EPG names as short as
possible, these settings are far more readable when presented to the vSphere administrator.
Activity Procedure
Navigate to: HOME → INVENTORY → HOSTS AND CLUSTERS

Select: Web1-TXX

Figure 143
On VM WEB1-TXX, select Summary.

If the VM is not already powered on, click the POWER ON option On VM WEB1-TXX:
Select EDIT SETTINGS...
Figure 144
Notice the Network is currently pointing at BLACK-HOLE
Highlight the network adapter and select the corresponding Network-Label (port- group) on the right
Network Label: TXX|WebApp|Web
Make sure the Device Status is:
CONNECTED (important!)
CONNECTED AT POWER ON (important!)
Click: OK to save

Figure 145
On VM WEB2-TXX, select Summary

If the VM is not already powered on, click the POWER ON option On VM WEB2-TXX,
select EDIT SETTINGS...

Figure 146
Highlight the network adapter and select the corresponding Network-Label (port- group)
on the right.
Network Label: TXX|WebApp|Web
Click: OK to save.

Figure 147
On VM DB TXX; Select: Summary

If the VM is not already powered on; Click: POWER ON
On VM DB-TXX; Select: EDIT SETTINGS...

Figure 148
Highlight the network adapter and select the corresponding Network-Label (port- group)
on the right
Network Label: TXX|WebApp|DB
Click: OK to save

Figure 149
Task 7: Verify VM Connectivity

So far, in Labs 2-4 we have configured our access policies, tied them to our tenant policies,
integrated our VMM and now have all of the devices attached to our fabric. At this point, we
should now see endpoints in our fabric and communication should be successful between
hosts in the same EPG/port-group. The final task of lab 4 is to verify that communication
between the VMs and our fabric is successful.
Activity Procedure
Open the vSphere client

To access the console of each VM: Select the VM (ex: WEB1-TXX)
Right-Click and choose OPEN CONSOLE from the menu. You can also click the
corresponding icon in the menu bar

Figure 150
Virtual Machine Credentials:
VMM Machine Credentials

VM Login VM Password VM SUDO Password
student lumos123 lumos123
First, ping the VM's default gateway to confirm connectivity to the ACI fabric by entering
PING 10.X.1.1 and pressing enter
Example (T10):
ping 10.10.1.1
Next, ping the WEB2-TXX from WEB1-TXX to validate connectivity between ESXi VMs
on the same port-group. (See table below)
VM IP Addressing
Tenant Web 1 IP Web 2 IP DB 1 IP
T01 10.1.1.11 10.1.1.12 10.1.2.11
T02 10.2.1.11 10.2.1.12 10.2.2.11
T03 10.3.1.11 10.3.1.12 10.3.2.11
T04 10.4.1.11 10.4.1.12 10.4.2.11
T05 10.5.1.11 10.5.1.12 10.5.2.11
T06 10.6.1.11 10.6.1.12 10.6.2.11
T07 10.7.1.11 10.7.1.12 10.7.2.11
T08 10.8.1.11 10.8.1.12 10.8.2.11
T09 10.9.1.11 10.9.1.12 10.9.2.11
T10 10.10.1.11 10.10.1.12 10.10.2.11

T11 10.11.1.11 10.11.1.12 10.11.2.11
T12 10.12.1.11 10.12.1.12 10.12.2.11
T13 10.13.1.11 10.13.1.12 10.13.2.11
T14 10.14.1.11 10.14.1.12 10.14.2.11
T15 10.15.1.11 10.15.1.12 10.15.2.11
T16 10.16.1.11 10.16.1.12 10.16.2.11
Finally, try to ping the DB-TXX virtual machine from WEB1-TXX. Does this final ping
work? Why or why not?
Figure 151

Communication Using Contracts
Table of Contents:
Task 1: Create and Assign Contracts for Default-Any
Task 2: Verify ICMP Connectivity between EPG2
Task 3: Verify MySQL Connectivity from Web to DB
Task 4: Contracts with specific port (ICMP and MySQL-3306)
Activity Objective
Within an End Point Group, all communication is permitted, however, by default, ACI is a
"white list" model -- this means that there is no communication between EPGs without it being
explicitly permitted. Contracts provide the means of permitting traffic between EPGs.
You can think of contracts as access lists. It is not a one to one match, but it helps to build up
the understanding of the concept. Contracts are comprised of three objects: Filters, Subjects
and the Contract itself. The contract object contains subjects and filters. So, a contract can
have multiple subjects and multiple filters.
After the contract is built it needs to be applied to the EPGs. To have traffic flow between
EPGs the contract needs to be associated with the EPGs intended to communicate. In this
lab we have two EPGs Web and DB, so the contracts we build will to have to applied to both
WEB and DB.
Contracts also have a provider consumer relationship. Meaning the flow of traffic is in one
direction. E.g. if we wanted the Web EPG to be able to SSH to the DB we would create an
SSH contract and apply it to DB as the provider and Web as the consumer.
In a whitelist model all traffic is denied by default between the security objects (EPGs). Only
traffic that the application needs to run is allowed through. In a traditional blacklist model all
traffic is allowed though until denied. For modern datacenters the preference is whitelist but
moving from a blacklist to whitelist model can be challenging.
So far, we have created two application EPGs. One for WEB and one for DB. All endpoints in
the WEB EPG should be able to ping each other and all endpoints in the DB EPG should be
able to ping each other. But WEB should not be able to ping DB and DB should not be able to
ping WEB. Also, both WEB and DB should be able to ping their respective default
gateways(subnets).
Note: You may notice that endpoints in different EPGs can ping other EPGs default
gateways(subnets). E.g. Web can ping DB’s subnet and DB can ping Web’s. This is normal for
subnets under the same VRF. Subnets cannot have policy(contracts) associated with them, so
they are open for communication.
Required Resources

provided below.
◦ Username: admin
APIC IP Address
https://10.203.254.24
RDP Connection
Tasks
Task 1: Create and Assign Contracts for Default-Any
Activity Procedure
By default, all VMs within an EPG should be able to communicate with each other, as
validated in the previous lab. In order to verify that inter-EPG communication via the ACI
fabric works as intended, we will define a contract to allow all traffic first in order to test.
Remember contracts have three parts: filters, subjects and the contract itself. We will start
with a filter and add an entry to allow all traffic. Next, we will create a contract and subject
then associate the filter with the subject. Finally, we will add the contract to our two EPGs,
Web and BD.
Contracts, subjects and filters are in the Security Policies folder in your Tenant.
Creating a filter
Filters are associated with contracts and contain entries like lines in an access list. A filter can
have multiple entries. Notice we are creating the filter first before the contract. This is fine as we
can associate the filter to the contract later. A filter can even be associated with multiple
contracts.
Expand: CONTRACTS
Right-click: FILTERS
Select: CREATE FILTER

Figure 152
Adding an entry
Entries have many options available. But our goal here is to allow all traffic (any/any).
Identity Name: TXX-default
Select: + to add the filter
Entry Name: TXX-default
Ether type: UNSPECIFIED
Select: UPDATE
Select: SUBMIT
Figure 153
Creating a contract

Next, create the contract to apply between our EPGs.
Right-click: CONTRACTS
Select: CREATE CONTRACTS
Figure 154
Name: TXX-default
Scope: GLOBAL
Click: + to add the subject to the contract

Figure 155
Setting the scope
This will determine where the contact can be applied. Setting it to global will allow us to share
this contract with other tenants in a future lab.
Creating a subject
Subjects also have many settings to tweak if required. QoS and DSCP can also be applied
here for example. We also set what kind of enforcement we want to set with “Apply Both
Directions” and “Reverse Filter Ports” by default traffic is allowed from consumer to provider
based on your filter. Also, the return traffic is allowed back from provider to consumer. “Apply
Both Directions” will automatically create an entry for the reverse traffic for you. While with
“Reverse Filter Ports” unchecked, you would have to specify what the return traffic looks like
with your own filter.
Name: TXX-default
Click: + to add a filter to the subject
Select: TXX/TXX-default from the drop down
Select: UPDATE
Click: OK to save the subject

Figure 156
Click: SUBMIT to save the contract

Figure 157
Assigning the contract
We will now assign the contract to our two EPGs. One with provider and one with consumer.
Make sure not to have a provider/provider or consumer/consumer. Always provider/consumer.
Note: Normally the consumer needs to initiate the connection to the provider. And only return
traffic is allowed back from the provider. With ICMP in ACI the provider can initiate an ICMP
request. For most other protocols to get bi-directional communication both EPGs would need
provide and consume the contract.
Expand: TXX → APPLICATION PROFILES → WebApp → Application EPGs → Web
Select: ADD PROVIDED CONTRACT

Figure 158
Contract Name: TXX/TXX-default


Figure 159
Next, we must consume (and also provide) the default service from another EPG, such as the
DB EPG.
Expand: TXX → APPLICATION PROFILES → WebApp → Application EPGs →
DB
Select: ADD CONSUMED CONTRACT
Figure 160

Contract Name: TXX/TXX-default
Figure 161
Task 2: Verify ICMP Connectivity between EPG2

With the contracts associated between EPGs, it is now time to test. The contract should allow
all traffic.
Activity Procedure
Access the Web (1, 2 or both) VM console from vSphere

Ping the DB VM from Web to confirm ICMP inter-EPG connectivity

Figure 162
Access the DB console from vSphere

Ping the Web VM from DB to confirm ICMP inter-EPG connectivity

Figure 163
Task 3: Verify MySQL Connectivity from Web to DB

Activity Procedure
Access the Web VM console from vSphere

From the Web VM, execute the following command and you should see "Connected to [DB
VM IP Address]"
telnet 10.XX.2.11 3306
Press Ctrl + ] and type "quit" to exit

Figure 164
Task 4: Contracts with specific port (ICMP and MySQL-3306)

Activity Procedure
Now that we have tested connectivity, remove the any/any contract from the EPGs and
create two new contracts; one for ICMP and one for SQL. This will resemble more of a
whitelist model.
Remove the any/any contract
First, we'll delete the contract from the Web and DB EPG.This will not delete the contract
from ACI overall just its association from the EPGs.
Expand: APPLICATION PROFILES → WebApp → Application EPGs → DB
Select: CONTRACTS
Right-click: TXX-default and DELETE

Figure 165
Select: YES to confirm

Figure 166
Repeat for the EPG Web and delete the contract.

Expand: APPLICATION PROFILES → WebApp → Application EPGs → WEB
Select: CONTRACTS
Right-click: TXX-default and DELETE

Figure 167
Select: YES to confirm

Figure 168
Try to ping from web1-TXX to db1-TXX or telnet the MySQL port 3306. They should now fail.
Figure 169
Create a specific filter for ICMP

Expand: CONTRACTS

Figure 170
Identity Name: TXX-ICMP

Entry Name: TXX-ICMP
Ethertype: IP
IP Protocol: ICMP
Select: UPDATE
Select: SUBMIT

Figure 171
Create a specific filter for MySQL

Expand: CONTRACTS

Figure 172
Identity Name: TXX-MySQL

Entry Name: TXX-MySQL
Ethertype: IP
IP Protocol: TCP
Destination: PORT RANGE 3306 - 3306
Select: UPDATE
Select: SUBMIT

Figure 173
Create the contract and subject for ICMP and add the ICMP filter

Figure 174
Name: TXX-ICMP
Scope: VRF

Figure 175
Name: TXX-ICMP
Select: TXX/TXX-ICMP from the drop down
Select: UPDATE
Select: OK to save the subject

Figure 176

Figure 177
Assigning the contracts
Expand: TXX → APPLICATION PROFILES → WebApp → Application EPGs → DB


Figure 178
Contract Name: TXX/TXX-ICMP

Figure 179

Go to the Web EPG to add a consumed contract
Figure 180
Contract Name: TXX/TXX-ICMP


Figure 181
Test to see if you can ping between the Web EPG and DB EPG
Figure 182
And from the DB EPG to the Web EPG

Figure 183
Create the contract and subject for MySQL and add the MySQL filter

Figure 184
Name: TXX-MySQL
Scope: VRF

Figure 185
Name: TXX-MySQL
Select: TXX/TXX-MySQL from the drop down
Select: UPDATE
Select: OK to save the subject

Figure 186

Figure 187
Assigning the contracts
Expand: TXX → APPLICATION PROFILES → WebApp → Application EPGs → DB


Figure 188
Contract Name: TXX/TXX-MySQL

Figure 189

Go to the Web EPG to add a consumed contract
Figure 190
Contract Name: TXX/TXX-MySQL


Figure 191
Testing communication form the Web EPG to the DB EPG
Now we will test port 3306 from a VM in the WEB EPG to a VM in the DB EPG.
Access the Web VM console from vSphere
From the Web VM, execute the following command and you should see "Connected to
[DB VM IP Address]"
telnet 10.XX.2.11 3306
Press Ctrl + ] and type "quit" to exit
Figure 192

Establishing Layer 3 External
Connectivity
Table of Contents:
Task 1: Build vPCs for Connectivity to Nexus 6K Switch
Task 2: Create Layer 3 External Routed Outside Connections
Task 3: Create OSPF Interface Protocol Policy
Task 4: Create Layer 3 Domain
Task 5: Create External Networks
Task 6: Verifying OSPF Neighbor Relationships
Task 7: Advertising Subnets from the ACI Fabric
Activity Objective
In this activity, students will configure Layer 3 Outside network relationships with routers
external to the ACI fabric. The L3Outs will be built using OSPF peering to a Nexus 6000
switch via vPC links. At the end of this lab students should understand the basic usage and
configuration of an L3Out, the different types profiles and how to verify successful route
peering with devices external to the ACI fabric.
To get routing to work in and out of the fabric we will need to set up some policies and
profiles.
In large ACI fabrics it is not practical, or desirable, to connect all external routers to all leaf
nodes. Instead, a common practice is to assign a pair of leaf switches as "border leaf" nodes.
Even though these leaf nodes are named something special ("border leaf switches") they are
no different from any other leaf in the fabric and function the same. The border leaf switches
run routing protocols and peer to the external devices, learning routes from outside of ACI,
and advertising routes to the outside world.
Leaf1 and Leaf2 are already cabled up to a pair of Nexus 6000 series switches, these nodes
will be the border leaf switches for the lab.
Each tenant will have a vPC formed to each of the upstream Nexus 6000 switches. Routing
will occur on an SVI between the fabric and the Nexus 6ks.
Required Resources

provided below.

◦ Username: admin
APIC IP Address
https://10.203.254.24
RDP Connection
Tasks
Task 1: Build vPCs for Connectivity to Nexus 6K Switch
Since we are attaching to new equipment (6001’s) we will set up more access policies.
We could use some of the same objects we used before like interface profiles but for this lab
we will create new objects. The objects that need to be unique are the interface policy groups.
One interface policy group represents one port-channel in ACI so each vPC will need its own
interface policy group and will require a LACP policy.
Activity Procedure
Create LACP Active Port Channel Policy.


Figure 193

Right-Click: PORT CHANNEL
Select: CREATE PORT CHANNEL POLICY

Figure 194
Name: TXX-LACP-Active
Mode: LACP ACTIVE
Click: SUBMIT

Figure 195
Leaf Policy Group’s/Interface Policy Groups.
We will need two interface policy groups, one for each vPC. Once again, even though the
interface policy groups will have the same policies. They still need to be unique -- one for each
vPC.
Create a N6K1 vPC Policy Group to represent the vPC to the N6K1.
Right-Click: VPC INTERFACE
Select: CREATE VPC POLICY GROUP

Figure 196
Name: TXX-vPC-N6K1
MCP Policy: DEFAULT
Port Channel Policy: TXX-LACP-Active
Click: SUBMIT

Figure 197
Create a N6K2 vPC Policy Group to represent the vPC to the N6K2.
Right-Click: VPC INTERFACE
Select: CREATE VPC POLICY GROUP

Figure 198
Name: TXX-vPC-N6K2
MCP Policy: DEFAULT
Port Channel Policy: TXX-LACP-Active
Click: SUBMIT

Figure 199
Review your work

Figure 200
Adding interfaces profiles
Since we have new connections, we will add in a new interface profile to connect to
the switch profile. Interface Profiles tell ACI what interfaces to configure on what leaf
switches. In the profile we will have two selectors, one for N6K1 and one for N6K2.
The table below lists what interfaces belong to what tenants and indicates what ports from the
201 and 202 leafs to the Nexus 6K1 and 6K2. You only want to add your leaf 201/202 ports to
you interface policy. The other side is there for reference and troubleshooting only.
Physical Connectivity
Tenant vPC Leaf 201 and 202 Ports N6K Ports
T01 vPC to N6K1 1/17 on both leaf's 1/1-2 on N6K-1

Create a vPC Leaf Interface Profile

Right-Click: PROFILES

Figure 201
Adding Interface Selectors
Name: TXX-vPC-L1L2
Select: + to add Interface Selector for N6K1

Figure 202
Name: TXX-N6K1
Interface ID: REFER TO TABLE 6.1
Interface Policy Group: TXX-vPC-N6K1
Click: OK

Figure 203
Select: + to add Interface Selector for N6K2

Figure 204
Name: TXX-N6K2
Interface ID: REFER TO TABLE 6.1
Interface Policy Group: TXX-vPC-N6K2
Click: OK

Figure 205
Click: SUBMIT

Figure 206
Create a Leaf Profile
Expand: SWITCHES → LEAF SWITCHES →PROFILES

Right-Click: PROFILES

Figure 207
Leaf Profile Name: TXX-vPC-L1L2-SP

Select: + to add LEAF SELCTOR
Leaf Selectors Name: TXX-vPC-L1L2-SS
Blocks: From the drop-down box → 201-202
Click: UPDATE

Figure 208
Click: NEXT to continue

Figure 209
Attaching the interface profile to the switch profile
We need to let ACI know which interfaces to configure on which switches. To do so,
we attach the interface profile to the switch profile and all ports in the interface profile
will be configured with the policies in the policy group on the switches in the switch
profile.
Select: Your tenants vPC interface profile
Click: FINISH

Figure 210
Verify the changes

Expand: SWITCHES → LEAF SWITCHES → PROFILES
Select: TXX-vPC-L1L2-SP

Figure 211
Task 2: Create Layer 3 External Routed Outside Connections

Routed outside networks are used to connect to outside the ACI fabric at layer 3. When you
build the objects used for the layer 3 connection you can think of it as going into the CLI of a
router or switch and building up the settings for routing. We are just making the changes via a
GUI instead of traditional command line.
Before creating an external routed outside you should already have planned out:
• Routing protocol you want to run and/or static routes.
• Any settings you would set in global configuration for the protocol, i.e. area ID for
OSPF etc...
• The domain to connect to. (ACI only) What VRF to attach to.
• What leaf switches to run the L3 Outside on. (border leaf switches)
• The VLAN’s needed if using SVI’s or routed sub interfaces.
• IP addresses for loopbacks and interfaces.
• What subnets you want to share back to the world.

Create a routed outside
Think of a routed outside as a single routing process with all the settings. You can have multiple
routed outsides per VRF and per tenant.
Expand: NETWORKING
Right-click: EXTERNAL ROUTED NETWORKS
Select: CREATE ROUTED OUTSIDE
Figure 212
Routing protocol and global settings
In the routed outside object, you set what protocol you want to use if any, the VRF to associate
it to the domain and any global settings.
Name: L3Out-OSPF
VRF: TXX/Main
Select: OSPF
Area ID: 0
OSFP Area Type: REGULAR AREA
Select: + under Nodes as Interfaces Protocol Profiles to create an entry

Figure 213
Creating the node profile
The node profile is where you choose the border leaf switches to use for the routing process.
Name: L3Out-OSPF-Nodes
Select: + under Nodes create the first entry

Figure 214
Add the first node

Node ID: 201
Router ID: 1.1.1.91
Click: OK

Figure 215
Select: + under Nodes create the second entry

Figure 216
Node ID: 202

Router ID: 1.1.1.92
Click: OK

Figure 217
Create interface profiles
Interface profiles determine what interfaces the protocol runs on. We have already set the
leaf switches, now we configure the settings on the interfaces for OSPF. There are three
options for layer 3 interfaces: routed interface, sub-interface and SVI’s. Since we are using
vPC for our links to the 6K’s we will need to build out two SVI interfaces to define the paths
for each VPC link. Each vPC will need an SVI for leaf 1 and leaf 2.
Select: + to create an OSPF Interface Profile

Figure 218
Name: L3Out-OSPF-Interfaces
Click: NEXT

Figure 219
In the next window we also leave everything default for now even though we will set the
OSPF policy later.
Click: NEXT

Figure 220
Create Interfaces
There are three options for layer 3 interfaces: routed interface, sub-interface and SVI’s. Each
option has its own benefits and complexities.
A routed interface allows for only one physical interface to be assigned to L3 out. It is also the
simplest to configure. Same as setting up a switch port for routing, by turning off switchport and
assigning it an IP address.
Routed sub-interfaces allow you to have multiple VRFs or multiple protocol processes sharing
the same physical interface. This is slightly more complex because we need to add VLANs
from a VLAN pool. You can think of this like a router on a stick configuration.
SVI’s are the most flexible because those links can be shared for layer three traffic and layer two
traffic. Also, you are able to use layer two load balancing using port-channels or vPC’s. More
complex because you need to assign VLAN’s and when using vPC assign VLAN’s and IP
addresses to both switches.
Here we will create two SVI interfaces one for each vPC. Create the first SVI:
Select: SVI Tab
Select: + to add the interface

Figure 221
Select the first vPC you created earlier in access policies.
If you don’t see it, go back to access policies and check it was configured correctly.
Path type: VIRTUAL PORT CHANNEL
Path: TXXvPC-N6K1

Figure 222
Assign a VLAN and IP address for the SVI.

Encap: vlan-2XX8
Side A IP address: 10.XX.5.254/24
Side B IP address: 10.XX.5.253/24
Click: OK

Figure 223
The Second SVI:

Select: SVI Tab
Select: + to add the interface

Figure 224
Select the second vPC you created earlier in access policies.
If you don’t see it, go back to access policies and check it was configured correctly.
Path type: VIRTUAL PORT CHANNEL
Path: TXXvPC-N6K2

Figure 225
Assign a VLAN and IP address for the SVI.

Encap: vlan-2XX8
Side A IP address: 10.XX.5.254/24
Side B IP address: 10.XX.5.253/24
Click: OK

Figure 226
Click: OK to save interface profile

Figure 227
Click: OK to save node profile

Figure 228
Click: NEXT
Figure 229
Click: FINISH

Figure 230
Task 3: Create OSPF Interface Protocol Policy

Activity Procedure
To prevent a mismatch in IP interface MTU settings from keeping OSPF neighbor

relationships from forming between ACI and the Nexus 6000 switches, we will need to create
and apply a protocol policy setting to sidestep this requirement.
Navigate to: TENANTS → POLICIES → PROTOCOL
Right-click: OSPF
Select: CREATE OSPF INTERFACE POLICY

Figure 231
Name: MTU-Ignore
Check: MTU-IGNORE
Click: SUBMIT

Figure 232
Apply the interface policy
Navigate to: Tenants → NETWORKING → EXTERNAL ROUTED NETWORKS →

L3Out-OSPF → LOGICAL NODE PROFILES → L3Out-OSPF-Nodes → LOGICAL
INTERFACE PROFILES → L3Out-OSPF-Interfaces → OSPF INTERFACE PROFILE
Drop down: MTU-IGNORE
Click: SUBMIT

Figure 233
Task 4: Create Layer 3 Domain

Activity Procedure
We've provided nearly all of the required information to ACI in regard to the OSPF L3-Out
configuration, but we haven't actually given it the ability to access and utilize the VLAN 2XX8
that we assigned. The AEP needs to be tied to L3Out-OSPF, which is done by creating a
Layer 3 Domain.
Expand: PHYSICAL AND EXTERNAL DOMAINS
Right-click: EXTERNAL ROUTED DOMAINS
Select: CREATE LAYER 3 DOMAIN

Figure 234
Name: TXX-L3Out-OSPF
Associated Attachable Entity Profile: TXX-AEP
VLAN Pool: TXX-VLANPool (dynamic)
Click: SUBMIT

Figure 235
Associate the domain to the external routed outside

Expand: EXTERNAL ROUTED NETWORKS
Select: L3Out-OSPF
In the right-hand window page:

Click: POLICY
External Routed Domain: TXX-L3Out-OSPF
Click: SUBMIT

Figure 236
Task 5: Create External Networks

Create External EPG's
At this point, all configuration for OSPF is ready but in ACI all interfaces need to be part of
EPGs to communicate. In the next section we will create the EPGs for the routed outside.
The EPGs that were created before classified endpoints based on VLAN tags. These network
EPGs will use IP addresses to classify endpoint. We will need to define two separate EPGs,
one for Dev and another for Users. This will allow us to provide different policy options for these
two outside networks.
If reviewing the OSPF neighbors on the Nexus 6Ks at this point, we still won't see any
neighbors established. Since we haven't defined any EPGs, there's no reason for the leaf to
implement the L3Out policy. For that, we'll need to define specific external networks.
Navigate to: TENANTS → TXX → NETWORKING → EXTERNAL ROUTED NETWORKS
→ L3Out-OSPF
Right-click: NETWORKS
Select: CREATE EXTERNAL NETWORK

Figure 237
Name: L3Out-EPG-Dev
Click: + to add entry

Figure 238
In the Dev EPG add the subnet of the Dev VM. This will allow only traffic with this subnet to
use the EPG.
IP Address: 10.XX.70.0/24
Select: SHARED SECURITY IMPORT SUBNET
Click: OK

Figure 239
Click: SUBMIT

Figure 240
Create another L3Out Network EPG for Users
In the User EPG add the subnet of the Users VM. This will allow only traffic with this subnet
to use the EPG.
Navigate to: TENANTS → TXX → NETWORKING → EXTERNAL ROUTED NETWORKS
→ L3Out-OSPF
Right-click: NETWORKS
Select: CREATE EXTERNAL NETWORK

Figure 241
Name: L3Out-EPG-Users
Click: + to add entry
Figure 242

Notice the subnet is set to 0.0.0.0/0. This represents everyone.
IP Address: 0.0.0.0/0
Select: SHARED IMPORT SECURITY SUBNET
Click: OK
Figure 243
Click: SUBMIT

Figure 244
Now we have two EPGs, one for Dev and one for Users. These are based on
subnet instead of VLAN tags and still allows us to apply policies based on their
EPG membership.
Task 6: Verifying OSPF Neighbor Relationships

We want to make sure OSPF is up and running correctly before we move on. Using Putty
(shortcut on the RDP desktop) you will first log into the 6K’s and check the OSPF neighbor
adjacency.
Activity Procedure
Login to: N6K1 and N6K2 using the information in the table below.
Nexus 6K Information
L3 Switch IP address Username Password
N6K1 10.203.254.27 admin lumos123
N6K2 10.203.254.28 admin lumos123
Please be gentle. Do not make any configuration changes on the Nexus 6K

switches.
On N6K1 run:
show ip ospf neighbor vrf TXX

Figure 245
You should now see your neighbor adjacency up and in a full/DR state. If you do not please
let your instructor know.
Now that you have a routing adjacency you can also see what routes have been exchanged
from your VRF in ACI.
On N6K2
show ip route ospf vrf TXX

Figure 246
You should notice that none of your tenant subnets are learned yet. 10.X.1.0/24 and
10.X.2.0/24. This is because we have not specified which subnets we want to
advertise yet. You can also view the routes from the ACI GUI.
Navigate to: TENANTS →TXX → NETWORKING → EXTERNAL ROUTED NETWORK
→ L3OUT- OSPF → LOGICAL NODE PROFILES → L3OUT-OSPF-NODES →
CONFIGURED NODES → TOPOLOGY/POD-1/NODE-201 →OSPF FOR VRF-TXX
MAIN → ROUTES
Figure 247

Task 7: Advertising Subnets from the ACI Fabric
Activity Procedure
OSPF neighbor relationships are established but routes are not being advertised from
ACI to the Nexus 6K's. For this to happen, we need to tell ACI that these subnets
should be publicly accessible, and that the bridge domain is associated with the newly
created L3Out-OSPF object.
Navigate to: Tenants → TXX → NETWORKING → BRIDGE DOMAINS
Select: Web
Select: POLICY
Select: L3 CONFIGURATIONS
Click: + to add a L3 Out to the BD
Select: TXX/L3Out-OSPF
Click: UPDATE
Figure 248
Navigate to: Tenants → TXX → NETWORKING → BRIDGE DOMAINS → Web →

SUBNETS
Select: 10.XX.1.1/24
Check: ADVERTISED EXTERNALLY
Check: SHARED BETWEEN VRF's
Click: SUBMIT

Figure 249
Repeat the above 2 steps to the DB bridge domain and subnet.

Navigate to: Tenants → TXX → NETWORKING → BRIDGE DOMAINS
Select: DB
Select: POLICY
Select: L3 CONFIGURATIONS
Click: + to add a L3 Out to the BD
Click: UPDATE
Figure 250
Navigate to: Tenants → TXX → NETWORKING → BRIDGE DOMAINS → DB →

SUBNETS
Select: 10.XX.2.1/24
Check: ADVERTISED EXTERNALLY
Check: SHARED BETWEEN VRF's
Click: SUBMIT
Figure 251
Login to: N6K1 and N6K2 using the information in the table below and check the
neighbor relationship and routing table again.
Nexus 6K Information
L3 Switch IP address Username Password
N6K1 10.203.254.27 admin lumos123
N6K2 10.203.254.28 admin lumos123
Please be gentle. Do not make any configuration changes on the Nexus 6K

switches.
Run the following command on N6K1:


Figure 252
On N6K2
Figure 253
You should notice your tenant subnets are now learned. 10.XX.1.0/24 and

10.XX.2.0/24 should be in your N6K routing tables.
You can also see what routes have been established in ACI.
Navigate to: NETWORKING → EXTERNAL ROUTED NETWORK → L3OUT- OSPF →
LOGICAL NODE PROFILES → L3OUT-OSPF-NODES → CONFIGURED NODES
→TOPOLOGY/POD-1/NODE-201 →OSPF FOR VRF-TXX MAIN → ROUTES
Figure 254

L3 Communication Using Contracts
Table of Contents:
Task 1: Create Protocol Filters
Task 2: Create Contracts
Task 3: Allow Communication Between External Networks and EPGs
Task 4: Allow Communication Between External Users and EPG Web
Task 5: Validate External Connectivity
reference table.
Activity Objective
In this activity, students will create contracts to permit traffic to be sent and received between
the internal ACI tenant EPGs and the External Network EPGs that reside outside the fabric.
The L3Out that was created in the last lab exercise will provide connectivity between the
fabric and the external EPGs. When finished this lab will provide students the basic
knowledge and workflow needed to establish connectivity between devices external to the
fabric and devices inside the fabric.
Required Resources

provided below.
◦ Username: admin
APIC IP Address
https://10.203.254.24
RDP Connection

Tasks
Task 1: Create Protocol Filters
Filters define the characteristics of the traffic we want to apply policy to. Things like
Ethertype, protocol and source and destination ports are defined in this filter.
Activity Procedure
Navigate to: TENANTS → TXX → CONTRACTS

Right Click: FILTERS
Figure 255
Filter Identity Name: TXX-L3Out-default

Click the + to add the Filter Entry
Filter Entry Name: TXX-L3Out-default
Ethertype: UNSPECIFIED
Click: UPDATE

Figure 256
Use caution when using the Tab key to auto-complete when adding filter entries, as this can
result in the inadvertent selection of a predefined named protocol. Always double-check the
accuracy of your Filter Entry before clicking Update.
Task 2: Create Contracts
Activity Procedure
In order to associate these traffic filter types with EPGs and Layer 3 constructs, they will need
to be bound to contracts and subjects.
Right Click: CONTRACTS
Select: CREATE CONTRACT

Figure 257
Contract Name: TXX-L3Out-default

Scope: GLOBAL
Click the + to add a Subject

Figure 258
Subject Name: TXX-L3Out-default

Click + to add a Filter Chain entry
Select: TXX/TXX-L3Out-default from the list of available choices
Click: UPDATE to save the Filter Chain entry
Click: OK to save

Figure 259
Figure 260

Task 3: Allow Communication Between External Networks and EPGs
Activity Procedure
The Contracts section for an L3 External Network EPG is in a different location when
compared to a 'normal' application EPG, but functionally the same.
We will apply the default contract.
First, we need to provide/consume these services from their respective EPGs.
Navigate to: TENANTS → TXX → APPLICATION PROFILES → WebApp →
APPLICATION EPGS → Web
Figure 261
Name: TXX/TXX-L3Out-default

Figure 262
Repeat the same steps for the DB EPG.

APPLICATION EPGS → DB

Figure 263
Name: TXX/TXX-L3Out-default
Figure 264

Now we need to apply the contract provide/consume to the L3 External EPG.
Navigate to: TENANTS → TXX → NETWORKING → EXTERNAL ROUTED
NETWORKS → L3Out-OSPF → NETWORKS
Select: L3Out-EPG-Dev
In the right pane, click CONTRACTS from the second row of tabs in the upper- right corner
Click + under PROVIDED CONTRACTS to assign a contract
Click the drop-down arrow to display the contracts available Select: TXX/TXX-L3Out-
default
Click: UPDATE to save
Figure 265
Repeat to add the "Consumed Contracts" to the L3 Out Network EPG.

Click + under CONSUMED CONTRACTS to assign a contract
Click the drop-down arrow to display the contracts available Select: TXX/TXX-L3Out-
default
Figure 266

Task 4: Allow Communication Between External Users and EPG Web
Activity Procedure
From all other external networks, we will create an undefined contract/filter to be used
between All external users and the EPG Web.
You could create specific filters/contracts to only allow HTTP and HTTPS services provided
by EPG Web, and ICMP from EPG Web only.
In order to allow these users to ping only EPG Web, we must create and provide/consume a
contract specific to these two objects.
Right click: FILTERS
Figure 267
Name: TXX-L3Out-users-default
Click + to add an entry
Ethertype: UNSPECIFIED
Select: UPDATE
Select: SUBMIT to save

Figure 268
Next, we will create a contract for that filter.

Right click: CONTRACTS
Select: CREATE CONTRACT
Figure 269

Scope: GLOBAL
Click + to create a Subject entry
Figure 270
Click: + to create a Filter Chain entry
Select: TXX/TXX-L3Out-users-default
Click: UPDATE to apply the Filter Chain entry
Click: OK to save

Figure 271

Figure 272
We will now add the provided contract to the Web EPG.

APPLICATION EPGS → Web
Right click: CONTRACTS

Figure 273
Figure 274

We will now add the contract to the L3Out Network EPG "L3Out-EPG-Users".
Navigate to: TENANTS → TXX → NETWORKING → EXTERNAL ROUTED
NETWORKS → L3Out-OSPF → NETWORKS
Select Network: L3Out-EPG-Users
In the right pane, click CONTRACTS from the second row of tabs in the upper- right corner
Click + under CONSUMED CONTRACTS to assign a contract
Click the drop-down arrow to display the contracts available
Figure 275
Task 5: Validate External Connectivity

At this point, we are ready to begin testing from the Dev and User VMs located outside of the
fabric.
Activity Procedure
In this lab, we will verify that hosts residing outside of the ACI fabric are able to communicate to
the VMs residing inside the ACI network. Follow these tasks to complete this lab.
Open the console to the Dev Virtual Machine in vCenter.
Under the cluster tenantXX-ext, right-click the VM dev-tXX
Select: OPEN CONSOLE

Figure 276
Log in using the credentials in the table below:
VM Guests
VM Login VM Password
student lumos123
Figure 277

Once logged in, click the Ubuntu icon in the upper-left corner
Then type TERMINAL in the search window
Click on the application TERMINAL
Figure 278
Attempt the following pings -- all should be successful

ping 10.XX.1.11
ping 10.XX.1.12
ping 10.XX.2.11
Figure 279
Attempt to SSH to one of the servers and login with the credentials provided

ssh student@10.XX.1.11
Figure 280
Close or minimize the terminal window Launch the Google Chrome web browser
Figure 281
Navigate to http://www.tXX.lumoscloud.com (or use the IP of your DB server)

Your results should be similar to that displayed below

Figure 282
Under the cluster tenantXX-ext, right-click the VM user-tXX

Select: OPEN CONSOLE
Figure 283
VM Guests
VM Login VM Password
student lumos123

Figure 284
Once logged in, click the Ubuntu icon in the upper-left corner Then type TERMINAL in the
search window
Click on the application TERMINAL
Figure 285
Attempt the following pings. You should only be able to reach the Web VM IP addresses.
Attempts to ping the DB server should FAIL
ping 10.XX.1.11

ping 10.XX.1.12
ping 10.XX.2.11
Figure 286
Close or minimize the terminal window Launch the Google Chrome web browser
Figure 287
Navigate to http://www.tXX.lumoscloud.com
Your results should be like that displayed below

Figure 288

Layer 2 External Connectivity
Table of Contents:
Task 1: Create the SPAN Application Profile and Related Objects
Task 2: Create Static Path Bindings
Task 3: Assign Physical Domain Association
Task 4: Enable L2 Unknown Unicast and ARPFlooding
Task 5: Assign VMM Domain Association
Task 6: Create ERSPAN Session and Capture Packets
reference table.
Activity Objective
In this activity, students will be establishing Layer 2 connectivity to a VLAN outside of the
fabric. Students will then be creating an ERSPAN destination to be utilized to monitor traffic.
At the end of this lab exercise students will have a basic understanding of configuration and
use of ERSPAN within ACI.
Required Resources

provided below.
◦ Username: admin
APIC IP Address
https://10.203.254.24
RDP Connection
Tasks
Task 1: Create the SPAN Application Profile and Related Objects
Activity Procedure
We are creating a new Application Profile. This is not an EPG that will be added to your
existing WebApp Application Profile.
Navigate to: TENANTS → TXX → APPLICATION PROFILE
Right-click: APPLICATION PROFILE

Select: CREATE APPLICATION PROFILE
Figure 289
In Lab 5, we manually created each Application Profile and EPG component. This
time let's save a few clicks and create these items using the built-in wizard.
Name: SPAN
Click: + to create an EPG
EPG Name: SPAN
Bridge Domain: CREATE BRIDGE DOMAIN

Figure 290
We will now create a new Bridge Domain.

Name: SPAN
VRF: TXX/Main
Select: NEXT

Figure 291
Click: + to add a Subnet entry
Figure 292

Gateway IP: 10.XX.3.4/24
Check the box for: ADVERTISED EXTERNALLY
Check the box for: SHARED BETWEEN VRFS
Select: OK to save
Figure 293
Click: + to add an Associated L3 Out

Click: UPDATE to save the entry
Click: NEXT

Figure 294

Figure 295
Back on the Create Application Profile Window

Name: SPAN
Bridge Domain: SPAN
Click: the drop-down menu for PROVIDED CONTRACT
Select: TXX-L3Out-default

Figure 296
We will now be adding the consumed contract:

Click: the drop-down menu for CONSUMED CONTRACT
Select: TXX-L3Out-default

Figure 297
Click: UPDATE

Figure 298
Your results should be like that displayed below.
Figure 299
Task 2: Create Static Path Bindings

Activity Procedure

We will need to create two static path bindings, one for each N6K to provide the desired
redundancy.
Navigate to: TENANTS → TXX → APPLICATION PROFILES → SPAN → APPLICATION
EPGS → SPAN
Right-click: STATIC PORTS
Select: DEPLOY STATIC EPG ON PC, VPC, OR INTERFACE
Figure 300
Path Type: VIRTUAL PORT CHANNEL

Path: TXX-vPC-N6K1

Figure 301
Encap: vlan-2XX9
Deployment Immediacy: IMMEDIATE
Mode: TRUNK

Figure 302
We will now create the second static port binding:

Navigate to: TENANTS → TXX → APPLICATION PROFILES → SPAN →
APPLICATION EPGS → SPAN
Right-click: STATIC PORTS
Select: DEPLOY STATIC EPG ON PC, VPC, OR INTERFACE

Figure 303
Path Type: VIRTUAL PORT CHANNEL

Path: TXX-vPC-N6K2
Encap: vlan-2XX9
Deployment Immediacy: IMMEDIATE
Mode: TRUNK

Figure 304
Your results should be comparable to that shown below.
Figure 305
Task 3: Assign Physical Domain Association

Activity Procedure
In the previous steps, we instructed the SPAN EPG to use VLAN 2XX9 to connect outside of
the fabric. However, the SPAN EPG does not currently have that VLAN assigned to it in its list

of resources. To do that, we need to find a way to associate SPAN EPG with an AEP that
references our VLAN pool. We accomplish this by adding a physical domain association.
Navigate to: TENANTS → TXX → APPLICATION PROFILES → SPAN →
APPLICATION EPGS → SPAN
Right-click: DOMAINS (VM AND BARE-METALS)
Select: ADD PHYSICAL DOMAIN ASSOCIATION
Figure 306
Physical Domain Profile: TXX-Physical


Figure 307
Task 4: Enable L2 Unknown Unicast and ARP Flooding

Activity Procedure
Since the SPAN EPG extends outside of the physical boundary of our ACI fabric, we will need
to change the default behavior by which ACI handles certain types of traffic; namely L2
unknown unicasts and ARP.
Navigate to: TENANTS → TXX → NETWORKING → BRIDGE DOMAIN
Select: SPAN
In the work pane (right-side) of the window:

Select: ‘POLICY’ tab from work pane
Select: FLOOD for "L2 Unknown Unicast"

Figure 308
A pop-up message will appear to provide a warning to make sure that "ARP
Flooding" must be enabled
Click: OK to continue
Figure 309
Verify: The ARP FLOODING checkbox should be checked, if it is not put a checkmark in
the box
Click: SUBMIT to save the changes to the Bridge Domain

Figure 310
Task 5: Assign VMM Domain Association

Activity Procedure
The SPAN virtual machine resides on the legacy infrastructure -- the Nexus 6000s. To provide
reachability from the fabric, where the virtual machine's default gateway lives, to the VM itself
we will need to extend the SPAN EPG out of the fabric into the Nexus 6000s. While doing this,
we will need to preserve the legacy VLAN numbering information so that the legacy equipment
does not have to be re-configured. This is very similar to many real-world migration scenarios --
ACI becomes the default gateway for a VLAN but needs to extend that VLAN/subnet outside of
the fabric to support hosts that have not yet been migrated.
Navigate to: TENANTS → TXX → APPLICATION PROFILE → SPAN → APPLICATION

EPGS → SPAN
Right-click: DOMAINS (VM AND BARE-METALS)

Figure 311

Resolution Immediacy: PRE-PROVISION
VLAN Mode: STATIC
Port Encap: vlan-2XX9

Figure 312
Task 6: Create ERSPAN Session and Capture Packets

Activity Procedure
IMPORTANT NOTE: There is a limit of 4 SPAN sessions per ALE based leaf (as of verified
scalability guide 3.0, 8 for LSE based leaf switches), because of this limitation not all students
will be able to capture SPAN data simultaneously. Please coordinate with students and the
instructor to ensure all students get a chance to capture data, and please ensure to disable
your SPAN session when complete!
Navigate to: TENANTS → TXX → POLICIES → TROUBLESHOOT → SPAN

Right-click: SPAN DESTINATION GROUPS
Select: CREATE SPAN DESTINATION GROUPS

Figure 313
Name: DEST-1
Destination EPG - Tenant: TXX
Destination EPG - Application Profile: SPAN
Destination EPG -EPG: SPAN
Destination IP: 10.XX.3.11
Source IP: 10.XX.3.4

Figure 314
We will now create a SPAN source group:

Navigate to: TENANTS → TXX → POLICIES → TROUBLESHOOT → SPAN
Right-click: SPAN SOURCE GROUPS
Select: CREATE SPAN SOURCE GROUP

Figure 315
Name: SPAN-SRC
Admin State: Enabled (default)
Destination Group: DEST-1
Click: + to create a Source entry

Figure 316
Name: Source1
Direction: BOTH (default)
Source EPG: uni/tn-TXX/ap-WebApp/epg-Web
Click: OK to save
Figure 317

Figure 318
From the VMWare vSphere application, open the console of the TXX-Span VM

Figure 319
VMM Machine Credentials

VM Login VM Password VM SUDO Password
student lumos123 lumos123

Figure 320
In the search window, type in "Terminal" Click on the application "Terminal" to open it
Figure 321
Attempt to ping the Virtual Machine's default gateway and the IP address of the ACI fabric on
this Bridge Domain, both should be successful
ping 10.XX.3.1
ping 10.XX.3.4

Figure 322
Minimize or close the terminal window

Launch Wireshark by clicking the icon on the left tray
Figure 323
To decode ERSPAN frames, we will need to change a setting within Wireshark preferences.
Click the edit preferences icon on the upper-right of the Wireshark window

Figure 324
Expand: PROTOCOLS
Figure 325
Start typing "ERSPAN" to jump to the ERSPAN protocol

Select: ERSPAN
Enable: FORCE to decode fake ERSPAN frame
Click: OK to save

Figure 326
Select the eth0 interface in the left pane of the Wireshark window Click: START
Figure 327
Allow the capture run for 30-60 seconds, then click STOP to end the capture Locate a
packet sourced from 10.XX.3.11 destined for 10.XX.3.4
Expand the packet properties and look for a section entitled Generic Routing
Encapsulation (ERSPAN), you may have to check more than one packet to find an
example
Observe the VLAN encapsulation of 2XX4, denoting that this packet was indeed sourced
by EPG Web

Figure 328
IMPORTANT NOTE: As there is a hardware limitation on the number of sessions that can be
running concurrently, disable your ERSPAN session once you have successfully verified its
operation.
Navigate to: TENANTS → TXX → POLICIES → TROUBLESHOOT → SPAN → SPAN
SOURCE GROUPS
Select: SPAN-SRC
Admin State: DISABLED
Click: SUBMIT to save the settings

Figure 329

ACI Fabric Discovery Guide

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

ACI Fabric Discovery Guide

Încărcat de

Drepturi de autor:

Formate disponibile

How to Use This Lab Guide

All student tenant numbers are indicated with XX – as in TXX-CDP-ENABLE –

Navigate to: FABRIC → ACCESS POLICIES

Select: CREATE VLAN POOL

Items in BOLD can only be changed by wiping the fabric:

Fabric name: [ACI Fabric 1]

Fabric ID: [1]

Number of controllers: [3]

You must have at least 3 APICs in a cluster. Up to 5 APICs are currently

Pod ID: [1]

Used for multi-pod.

Standby controller: [Y]

Standby Controller ID:

Controller ID: [1]

Controller name: [apic1]

TEP address pool: [10.0.0.0/16]

INFRA VLAN: [4]

GIPO multicast pool: [225.0.0.0/15]

Enable IPv6 management: [N] IPv4 management IP:

IPv4 gateway IP:

OOB interface default gateway.

Management speed/duplex: [auto]

Enable strong passwords: [Y]

Task 2: Initial ACI Fabric Configuration

Review the configuration for accuracy

Click: Proceed to (IP address)

If applicable, Click: NO on the warning message

The System Dashboard is now displayed. The Dashboard provides a single-pane-of-glass

Now start examining the physical switches in the fabric.

Select: INTERFACE in the right-hand pane

Select: POD 1 from the left-hand pane

Task 3: Fabric Pod and Access Policies

Task 4: Out-of-Band (OOB) Management

In the Create Static Node Management Addresses pop-up window:

Name: Add the new Switch IDs in the field

Task 5: APIC GUI Overview

Click: L4-L7 SERVICES in the top menu as shown below

Click: VIRTUAL NETWORKING in the top menu

Click: FABRIC in the top menu

You should already be familiar with the INVENTORY sub-menu

Familiarize yourself with the other two sub menu items.

Click: ACCESS POLICIES

Click: DASHBOARD on the sub-menu

Click: CONTROLLERS on the sub-menu

Allocation Mode: DYNAMIC ALLOCATION

Enter your Tenant's VLAN range (see table)

Add in the VLAN Range: 2XX5 to 2XX9

Select SUBMIT to save the VLAN Pool

Task 2: Create a Physical Domain

Task 3: Create an Attachable Access Entity Profile

Do not make any changes to the next screen

Task 4: Create the Interface Policies

In production deployments, we recommend that the interface policy naming convention

To create the CDP Interface Policies for your tenant:

Repeat the above process to create a CDP Disabled policy.

To create the LLDP Policies:

Repeat the above process to create a LLDP Disabled policy.

Create MAC-Pinning Policies:

Task 5: Create the Interface Policy Groups

Navigate to: FABRIC → ACCESS POLICIES

Repeat the process for FI-B:

Click: SUBMIT to save