BRKACI-2300-ACI For VMware Admins

#CLUS
ACI for VMware

Admins
VMM, unicorn overlays and alike
Nicolas Vermande, Technical Marketing Engineer

@nvermande
BRKACI-2300
#CLUS
Agenda
• An orchestration tool for vCenter
• Entering the Goldilocks Zone
• Microsegment all the things!
• Containers as First-Class Citizens
• Overlays Inception?
• Extending the Virtual Datacenter
#CLUS BRKACI-2300 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session SPEAKER 1
How SPEAKER 2
1 Find this session in the Cisco Live Mobile App WEBEX TEAMS
2 Click “Join the Discussion”

3 Install Webex Teams or go directly to the team space
DOCUMENTS
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKACI-2300

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Objectives of this session
• Understand the value of ACI in a VMware
environment
• Have an overview of the solutions part of the ACI

portfolio that can help VMware admins solve real
challenges
• Describe good practices when interoperability

between VMware solutions and ACI is required
• Map Cloud Native solutions hosted in a vSphere

environment to ACI extended architecture
Cisco ACI is a versatile
solution to address network
management AND SDN
challenges
Focus for this session
ACI domains define
where and how to
deploy policies
And it can also
orchestrate vCenter
port-groups creation
Virtual Machine Manager Domain
Control Plane
VMM Domain
• The VMware Virtual Machine Manager Domain (VMM) VLAN 1001 VLAN 2030
defines a relationship between APIC and vCenter
• Each VMM maps to a VDS that is configured on vCenter
(provisioned through vSphere APIs)
EPG EPG
Configured
C Policy
• Each EPG maps to a port-group with dynamic VLAN Web App
allocation
Resolved Policy
• APIC reports full vCenter inventory
dot1q trunk
(1001,2030)
• Host Teaming and Failover Policy is automatically configured ESXi Host
• Policies are deployed on-demand. (other option is VM# VM# VM# CDP/LLDP enabled
immediate)
VM# VM# VM#
• Enables the use of ACI vCenter Plugin
Resolution vs Instrumentation Immediacy
• Resolution Immediacy – When is policy downloaded?
• Immediate: When hypervisor attached to VDS Policy = VRF/BD/EPG + ACL
• On-Demand: When VM is attached to Port-group
• Pre-provision: Not relying on LLDP, based on AAEP (solves chicken and egg
problem for vmkernel ports)
• NO-EFFECT on Physical Domain (always resolved with AAEP)
Resolution vs Instrumentation Immediacy
• Deployment Immediacy – When is policy implemented in TCAM?
• Immediate: As soon as policy is downloaded
• On-demand: When first packet hits the leaf
Enhanced LACP Policy
• With ACI 4.0(1h), APIC can provision enhanced LACP policies in VDS
• Multiple LAGs can be created on a VDS
• APIC asks user to choose LAG uplink during EPG association to VMM domain
Choose load-balancing mode
Choose # of uplinks
Choose LAG policy created
Result in vCenter
ACI can also control
host network
configuration
Use vSwitch Policies when host is not directly connected
Attachable Access
Entity Profile vPC - PC - Access
Interface Interface
AAEP Policy Profile /
Group Selector
ESX Teaming and Failover policy is

VMM determined based on this
information
What happens if host is not directly
connected???
AVE is operating in
user space, terminating
VXLAN tunnel in ESXi
The Goldilocks Zone
Getting closer to the Hypervisor
• Closer to the workloads
• Introduces new capabilities

• VXLAN termination on HV
• Software-only Overlay
• Connection tracking
• Micro-Segmentation
FTEP 10.0.0.32
• Local switching VXLAN tunnel
AVE Architecture
Port-groups mapped to Isolated
PVLAN based on EPG configuration vmkernel port i/o (management,
forcing inter-VM E-W traffic via AVE Physical NICs vMotion, NFS, etc) does not transit
AVE
VDS
Isolated PVLANs
vmkernel
ports
VM VM VM
AVE
Outside trunk
Inside trunk for traffic to/from the ACI fabric
configured in Promiscuous Mode with Configured with infra VLAN or APIC
Primary and Secondary VLANs VLAN pool depending upon AVE
mode (VXLAN/VLAN)
AVE Architecture
AVE Scheduler User space scheduler is similar to kernel

DPDK mode scheduler
AVE receives on Secondary VLAN and

Kernel forwards on Primary VLAN
vmxnet3 vmxnet3 vmxnet3 Support VXLAN and VLAN
VLAN-10 VLAN-20
Internal External Mgmt
VDS
AVE Architecture
AVE SchedulerExternal Port-group can be

DPDK User space scheduler is similar to kernel
backed by VLAN Pool or ACI mode scheduler
Infra VLAN
AVE receives on Secondary VLAN and
Kernel forwards on Primary VLAN
vmxnet3 vmxnet3 vmxnet3 Support VXLAN and VLAN
VLAN-10 VLAN-20
Internal External Mgmt
VDS
VM sends traffic in
Internal AVE Port-group Secondary
is Promiscuous Trunk Isolated VLAN
How do we scale performance
Management
• DPDK for packet processing

ESXi
AVE-SVM External
• Multi-VTEPs VXLAN load-balancing
Internal
Promiscuous Trunk VTEP1 VTEP2 VXLAN (x2) or external VLAN (x1)

• 2x VXLAN NICs and 1x VLAN NIC (50-75) and (76-100)
vmnic
VDS
• Multiple internal links for PVLAN
uplink
VXLAN / VLAN
distribution (P,S)=50,51 (P,S)=52,53
encapsulation
• Evenly split promiscuous trunks
How do we scale performance
Management
Each VTEP is linked to

a single active VMNIC
• DPDK for packet processing
ESXi (other is standby)
AVE-SVM External
• Multi-VTEPs VXLAN load-balancing
Internal
Promiscuous Trunk VTEP1 VTEP2 VXLAN (x2) or external VLAN (x1)

• 2x VXLAN NICs and 1x VLAN NIC (50-75) and (76-100)
vmnic
VDS
• Multiple internal links for PVLAN
uplink
VXLAN / VLAN
distribution Traffic between
(P,S)=50,51
local (P,S)=52,53 encapsulation
• Evenly split promiscuous trunks VMs and AVE is evenly

split between 2 vNICs
AVE Native (by-pass) mode
VMKernel AVE-SVM
(eg. Storage, Mgmt)
inside outside
vmnic
uplink
VDS
Pro-active HA
vCenter
• Health Provider defined for AVE
VMM
DRS Cluster
• User can trigger AVE maintenance
mode Ø
Ø
Maintenance Mode
Quarantine ESXi ESXi ESXi
• Set host is yellow state Health Provider AVE AVE AVE
• When host maintenance mode is
VM VM VM VM
enabled, AVE is powered off after host
evacuation
• AVE is powered on after host is up AVE VMM Domain
• Host is kept in yellow state while opflex

is not up APIC
Microsegment all
the Things!
Reduce attack surface of ESXi hosts
10.10.10.0/16
SSH Ext EPG

EPG NFS
vSphere Client Consumer Server
DHCP
Provider
….
NFS Server
Provider Consumer
EPG EPG EPG NFS

vMotion mgmt Client
intra-EPG isolation
Simplify network by flattening IP subnets
Contract Contract
Cluster A Cluster 02
Cluster B
vMotion Network vMotion Network
Mgmt Network Mgmt Network
vMotion Subnet: 192.168.100.0/24
Management Subnet: 192.168.200.0/24
EPG classification can leverage VM attributes
• 𝜇EPG defines a security zone that includes VMs with common attribute set
VLAN 1500 / VXLAN 346500 EPG
TEST
VM# VM# VM# VM# VM# VM# VM# VM# VM#
𝜇EPG 𝜇EPG 𝜇EPG

Zone = Web Zone = DB
Web App DB internet
Zone = App
• IPS can place infected VMs into isolated containers
VLAN 1500 / VXLAN 346500

Service Insertion
EPG
PROD
VM# VM# VM# VM# VM# VM# VM# VM# VM# Ext EPG
𝜇EPG Remediation
Quarantine = True infected ports
𝜇Seg is available with VDS, AVS and AVE
1
• Supported with AVE (ACI 3.1) Allow 𝜇Seg
2
• Supported with VDS and EX/FX 4
based leaf. (ACI >= 1.3) Proxy-ARP
Steps:
1. APIC connects to vCenter and fetches VM
inventory including the attributes. Any changes
in VM attribute are synced based on VC events.
2. When user configures “EPG PROD” with ‘Allow
VDS
Micro-Segmentation’, APIC pushes it as
isolated-PVLAN based port-group to steer
traffic to the leaf (P:100,S:200)
3. VMs attached to the port-group are pushed to
the leaf as mac EPG
EPG
4. User creates a new uSeg EPG with Attributes
PROD VM# VM# VM# 3
5. APIC does the attribute matching to MAC-list 𝜇EPG Web
6. APIC updates MAC-list to uSeg EPG on the leaf
MAC = A, B,
B C Zone = Web
5 MAC = C
ACI Security tool set
• vzAny: EPG shortcut representing all Endpoints within a VRF (including external L3
Out EPG)
• Preferred Group: Group of EPGs that can communicate without any contract
• vzAny and Preferred group are mutually exclusive
• Contract Inheritance: User can compose new EPG contracts from parents EPGs. All
corresponding contracts are associated to the new EPG.
• Any modification on parent EPG affects all children
• Intra-EPG Contract: contract is applied within an EPG
• Contract Blacklist: Adds deny filters to contract
Consuming SDN
API from vCenter
ACI vCenter plugin
• Stateless, does not store any vCenter Plugin

information: fetch everything
VMM Domain
from APIC
• VMM must already exist
vSphere Web Client
ACI vCenter plugin
No in-depth Create EPGs, Implement Insert L4-7

knowledge of subnets and distributed Service
ACI required default gateways security
Automatic VLAN creation and network stitching for Service Insertion
Network is still under control
CRUD Operations External Connectivity, Troubleshooting
• Can configure, read, update or delete: • Limited Operations on L2/L3Outs

• Tenant • Can consume existing external EPGs
• Application Profile • Can’t create, edit, delete
• EPG / MicroEPG • L4-7 Service Graphs
• Contract • Can use existing Service Graph
• Filter • Can’t create Service Graph template
• VRF • Can edit empty mandatory
parameters of a function profile
• Bridge Domain
• Troubleshooting Tools
Now let’s move to
the cloud!
vRO/vRA plugin for ACI
Service Blueprints
• Day 1 Operations
Event Broker Subscription Integration
vRA Blueprint
payload
Event triggered Contextual variables
(provisioning, (OS, system generated
decommissioning) variables, custom variables)
Events
RabbitMQ Message Bus
Subscriptions
Execute workflow
upon event vRO
Subscription to ACI
plugin blueprint events
payload
Containers as
First-class Citizens
Containers in VMs?
Management tools:
- Change management granularity
- Single Management Interface for VMs and
container hosts across multiple locations
(centralized SSO,vCenter Templates)
- Take advantage of vSphere high-availability and
resource scheduling capabilities (HA, DRS)
Security
- VM encapsulation as logical boundary
- Better isolation
Storage
- HCI integration (Hyperflex)
- Storage optimization (SIOC, storage DRS)
ACI and K8S Integration Deployment Architecture
• Integration supported for K8S nodes as bare metal host or VM
ACI VMware VMM

0 Check pre-req
Provision 1 VDS 3
VMware VMM OOB Trunk port-group
(auto-created) Objects get
VTEP
created in APIC
VTEP
VXLAN
2 Container Host
Infra VLAN
ACI CNI Plugin
Deploy CNI Plugin Pod Subnet
- Container Controller External Service Subnet
- Host agent + Opflex Node Service Subnet
- OVS
Overlays Inception
Why Running Software Overlays over ACI?
• ACI is the best transport from a fabric connectivity and

network management perspective
• Some location may not have ACI-based equipment (vPOD
can help here!)
• It may not be possible to dissociate the overlay from a
particular solution (older Docker version for Swarm,
VMware vCloud Director etc)
• The software overlay was “already there”
ACI provides mgmt-
How about VMware NSX-V? plane visibility and
adds L3 capabilities
NSXv Mgr NSXv Ctrl

HW VTEP
VXLAN Overlay HW VTEP
Network
BM
Virtualization ToR ToR ToR ToR
ACI can still do VXLAN

ESXi ESXi ESXi
Service
Insertion Network
Services VPN NAT SLB Perimeter Firewall
ACI provides
the overlay
Security
+ Ecosystem Partners
Web HTTPS App 3306 DB
Option 1
• Use Micro-segmentation and Network Services:
• No need for Controllers, Edge Gateways and Edge Racks
• Substantial savings for compute resources
• No connectivity island
• Dedicated Security API, NSX security tags can be for automation
• NSX network services can be provisioned on demand: SLB or NAT, FW
• Take advantage of ACI policies virtual, physical and containers domain knowledge
• Single API shared across multiple teams to orchestrate application deployment and
infrastructure
• E/W stateful security for VMs, while ACI brings service insertion capabilities for N/S
Architecture example
Redirect HTTP/HTTPS Permit ANY
Ext EPG C EPG Web C EPG App
VRF CTX-01
10.10.1.0/24 192.168.1.0/24
L3 Out
.1 .1
Ext EPG EPG Web
VIP: 10.10.1.100
Permit
BD-WEB Any
192.168.2.0/24
BD-ESG SLB
.1
On-demand EPG App
load-balancing PBR – Service Graph
BD-App
Shadow EPG gets automatically
created with corresponding port-
group
Option 2
• ACI as the underlay and L3 boundary
• All VTEPs can be part of the same subnet
• ACI can further provide VTEP subnet segmentation with appropriate EPG mapping
EPG A EPG B
TZ – Cluster A TZ – Cluster B
BD 10.30.0.1/16
S1 S2 SVI or
L2 ext Subinterface
L2 ext Core
Benefits vs non-ACI L3 Fabrics*
WAN/
DCI
• No need for Edge Racks:
L1 L2 L3 L4 L5 L6 L7 L8 o Perimeter ESG for Tenant/Customer is
part of the tenant
o Edge physical failure domain is
independent from other tenants
VLAN
ESG
VXLAN • No L2 isolation at ToR for non-VM traffic:
o ACI provides L2 reachability between
DLR
Customers or Tenant racks.
o ESXi hosts network configuration is
drastically simplified. (No need for
North/South Flow multiple VMKernel TCP/IP Stacks)
Tenant A (Rack 1-2) Tenant B (Rack 3-4)
• Enhanced Security with AAEP, Security

VNI 5001 VNI 6001 Domains and fabric white-list model
VM# VM# VM# VM# VM# VM#
VNI 5002 VNI 6002

VM# VM# VM# VM# VM# VM#
*Example shows tenants limited to specific racks.

ACI Peering with NSX Edge Gateway
• ACI as a transit network
• Adding edge gateways doesn’t impact WAN/Core routers
• ACI as a replacement for second tier Edge Gateway
• Reduce routing table size advertised to Edge Gateways
ACI is now used as

ACI used as an L2 transit routing, between
network 2 L3 out constructs
ECMP
Stateful functions (FW) Stateful functions (FW)
HA pair HA pair
ACI Integration with NAT VNF
Permit HTTP/HTTPS
Ext EPG C EPG NAT
VRF CTX-01
10.10.1.1/24 BD-ESG VXLAN / Geneve
L3 Out
Ext EPG EPG NAT

Data-plane invisible
for ACI
No need for extra

virtual routing layer*
EPG VTEP
BD-VTEP
10.10.2.1/24
*You can’t have ESG with NAT + ECMP: another routing layer is required for
ECMP
ACI Peering with Virtual Router VNF
Leaf 101 Leaf 102 Leaf 103 Leaf 104
Routing adjacency
L3Out
Host 1 Host 2 Host 3 Host 4
VMware
VM#
DRS Cluster
Fabric-wide MAC: 0022.bdf8.19ff

Routing occurs
at the directly Selected NH
connected ToR L3Out
VMware
VM#
DRS Cluster
Routing occurs
at the directly
connected ToR
Selected NH
L3Out
VMware
VM#
DRS Cluster
New Model for better
integration with VNF:
Floating L3 out and

VMM integration
Floating IP for L3 Out
L3Out
Routing adjacency
VM# VM#
External network
prefixes
BD needs to be
stretched to
roaming border leaf
Routing adjacency
L3Out
VM# VM# VM#
External network
prefixes
Floating IP* required so
transit subnet can be
deployed on leaf
192.168.10.1/24 192.168.10.2/24 FIP: 192.168.10.250/24 FIP: 192.168.10.250/24
L3Out
VM# VM#
External network *Floating IP is a “dummy” IP that is not used for data-plane

prefixes or routing protocols
South-North sample flow
L3Out
VM# EPG VM# VM#
External network
prefixes
North-South sample flow
L3Out
VM# VM# VM# EPG
External network
prefixes
Extending the
Virtual Datacenter
High Availability vs Disaster Recovery
• HA provides non-orchestrated recovery within a single management domain
• Single failure domain
• Zero RPO can be achieved via synchronous storage replication
• RTO can be minimized
• vSphere HA is responsible for recovering workloads
• Depending on recovery units (e.g. Network, Site, Application), L2 extension and/or flooding
may be required
High Availability vs Disaster Recovery
• DR provides recovery processes and orchestration across distinct management
domains
• Multiple failure domains
• Long distance can rule out synchronous replication
• RTO generally takes longer to achieve (human decision to activate Recovery Plan)
High Availability with vMSC
• HA across locations is achieved via vSphere Stretched Clusters (vMSC)
vSphere vMSC Requirements
• Stretched Storage (uniform or non-uniform access)
• 10 ms RTT over management network and synchronous storage replication
• 150 ms RTT for vMotion or Storage vMotion
• 250 Mbps per concurrent vMotion session

• Single storage subsystem
• Check the following whitepaper for more details:

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-vsphere-metro-
storage-cluster-recommended-practices-white-paper.pdf
Disaster Recovery with SRM
• VMware Site Recovery Manager enables orchestrated recovery of VM workloads across 2
vCenter
SRM Requirements
• Supported SRA (Storage Replication Adapter) or vSphere Replication
• Protections Groups defined
• Network reachability
• By default, IP addresses are not changed

• Primary/Backup prefix advertisements
• L2 DCI
• Manual routing adjustment
• IP can be changed via VMware tools and customization script

• Dynamic DNS updates
• DNS update scripts
HA with ACI (Multipod)
Inter Pod
Network
L3 Pod 1 Pod 2
L3
Out Out
MP-BGP VXLAN EVPN
GW GW GW GW GW GW GW GW GW GW
APIC Cluster
vCenter
VDS
Port-group EPG VM VM VM
Port-group EPG VM VM VM VM VM VM
Port-group EPG VM VM VM VM VM VM
Stretched EPGs
ACI Multipod Properties and Benefits
• Distributed GW across locations
• Local exit point always preferred with IGP metric

• Ingress can be addressed by GOLF or ACI Host Based Routing
• Contract locally enforced (if destination is known)
• Service Insertion with PBR

• Active/Active or Active/Standby
• Single vCenter / Single VMM VDS
• Single APIC cluster
• 12 Pods
• 50ms RTT
Disaster Recovery with ACI (Multisite)
Inter Site
Network
L3 Site 1 Site 2
L3
Out Out
MP-BGP VXLAN EVPN
GW GW GW GW GW Per BD selective GW GW GW GW GW
flooding
vCenter Multisite Orchestrator vCenter
VDS VDS
Port-group EPG VM VM Port-group EPG
Port-group EPG VM VM Port-group EPG VM VM
Port-group EPG VM VM Port-group EPG VM VM
ACI Multisite Properties and Benefits
• Single policy model across multiple locations and ACI fabrics (12 Sites)
• Multiple failure domains
• Can select L2 flooding per BD
• Up to 150 msec RTT latency supported between MSO nodes
• Higher latency (500ms to 1s RTT) between MSO nodes and managed APIC clusters
• Support for cross-vCenter vMotion without flooding
• MSO manages Fabric Virtualization policies
• Simple public REST API to create policies, push to multiple sites and maintain synchronization
• Same port-group names at protected and recovery sites
• VMM domain properties and EPG association are managed per fabric
Key Takeaways
ACI Overlay and VMware solutions
• ACI provides the best overlay manager for VMware based solutions
• APIC is tightly integrated with VMware VDS and allows for flexible network designs
• ACI REST API allows for simple integration by means of plugins maintained by Cisco
(available on CCO)
• VMware vCenter
• vRealize Automation
• Network team keeps CONTROL over the physical AND the virtual network
• ACI accelerates VM provisioning and lifecycle management across multiple locations

without compromise on security and connectivity
Q&A
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Thank you
#CLUS
#CLUS

BRKACI-2300-ACI For VMware Admins

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

BRKACI-2300-ACI For VMware Admins

Încărcat de

Drepturi de autor:

Formate disponibile

#CLUS

ACI for VMware

Nicolas Vermande, Technical Marketing Engineer

2 Click “Join the Discussion”

4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKACI-2300

• Have an overview of the solutions part of the ACI

• Describe good practices when interoperability

• Map Cloud Native solutions hosted in a vSphere

Choose load-balancing mode

Choose LAG policy created

ESX Teaming and Failover policy is

• Introduces new capabilities

AVE Scheduler User space scheduler is similar to kernel

AVE receives on Secondary VLAN and

vmxnet3 vmxnet3 vmxnet3 Support VXLAN and VLAN

AVE SchedulerExternal Port-group can be

vmxnet3 vmxnet3 vmxnet3 Support VXLAN and VLAN

• DPDK for packet processing

Promiscuous Trunk VTEP1 VTEP2 VXLAN (x2) or external VLAN (x1)

• Evenly split promiscuous trunks

Each VTEP is linked to

Promiscuous Trunk VTEP1 VTEP2 VXLAN (x2) or external VLAN (x1)

• Evenly split promiscuous trunks VMs and AVE is evenly

• Host is kept in yellow state while opflex

SSH Ext EPG

EPG EPG EPG NFS

vMotion Network vMotion Network

Mgmt Network Mgmt Network

vMotion Subnet: 192.168.100.0/24

Management Subnet: 192.168.200.0/24

𝜇EPG 𝜇EPG 𝜇EPG

• IPS can place infected VMs into isolated containers

VLAN 1500 / VXLAN 346500

• Intra-EPG Contract: contract is applied within an EPG

• Contract Blacklist: Adds deny filters to contract

• Stateless, does not store any vCenter Plugin

vSphere Web Client

No in-depth Create EPGs, Implement Insert L4-7

Automatic VLAN creation and network stitching for Service Insertion

CRUD Operations External Connectivity, Troubleshooting

• Can configure, read, update or delete: • Limited Operations on L2/L3Outs

ACI VMware VMM

• ACI is the best transport from a fabric connectivity and

NSXv Mgr NSXv Ctrl

ACI can still do VXLAN

Ext EPG C EPG Web C EPG App

• Enhanced Security with AAEP, Security

VNI 5002 VNI 6002

*Example shows tenants limited to specific racks.

ACI is now used as

Stateful functions (FW) Stateful functions (FW)

Ext EPG C EPG NAT

Ext EPG EPG NAT

No need for extra

Leaf 101 Leaf 102 Leaf 103 Leaf 104

Fabric-wide MAC: 0022.bdf8.19ff

Host 1 Host 2 Host 3 Host 4

Leaf 101 Leaf 102 Leaf 103 Leaf 104

Floating L3 out and

Leaf 101 Leaf 102 Leaf 103 Leaf 104

Host 1 Host 2 Host 3 Host 4

Leaf 101 Leaf 102 Leaf 103 Leaf 104