e-dokumen.com

Support for analytics: The newest leaf switches and spine switch line cards provide flow measurementcapabilities for the purposes of analytics and application dependency mappings. Support for link-level encryption: The newest leaf switches and spine switch line cards provide linerate MAC security (MACsec) encryption. Scale for endpoints: One of the major features of Cisco ACI is the endpoint database, which maintainsthe information about which endpoint is mapped to which Virtual Extensible LAN (VXLAN) tunnelendpoint (VTEP), in which bridge domain, and so on. Fibre Channel (FC) and Fibre Channel over Ethernet (FCoE): Depending on the leaf model, you canattach FC and/or FCoE-capable endpoints and use the leaf switch as an FCoE NPV device. Support for Layer 4 to Layer 7 service redirect: The Layer 4 to Layer 7 service graph is a feature thathas been available since the first release of Cisco ACI, and it works on all leaf switches. The Layer 4 toLayer 7 service graph redirect option allows redirection of traffic to Layer 4 to Layer 7 devices basedon protocols. Microsegmentation, or EPG classification capabilities: Microsegmentation refers to the capability toisolate traffic within an EPG (a function similar or equivalent to the private VLAN function) and tosegment traffic based on virtual machine properties, IP address, MAC address, and so on. Ability to change the allocation of hardware resources, such as to support more Longest Prefix Matchentries, or more policy CAM entries, or more IPv4 entries. This concept is called "tile profiles," and itwas introduced in Cisco ACI 3.0. For more information, see the following tches/datacenter/aci/apic/sw/kb/b Cisco APIC Forwarding Scale Profile Policy.pdf. You may also want to read the Verified Scalability .html#Verified Scalability Guides.For more information about the differences between the Cisco Nexus 9000 series switches, see the followingdocuments: 38259.html e switchesThe spine switches are available in several form factors both for modular switches as well as for fixed formfactors. Cisco ACI leaf switches with name ending in -GX have hardware that can operate both as spine and asleaf. At the time of this writing some -GX leaf switches can only be installed with the Cisco ACI leaf switchsoftware and some can only be installed with the spine switch software.The differences among spine switches with different hardware are as follows: Port speeds Support for analytics: although this capability is primarily a leaf switch function and it may not benecessary in the spine switch, in the future there may be features that use this capability in the spineswitch.Support for link-level encryption and for tches/datacenter/aci/aci multi- 2022 Cisco and/or its affiliates. All rights reserved.Page 6 of 218

ion-Guide-201 chapter 011.html#id 79312. Support for Cisco ACI Multi-Pod and Cisco ACI Multi-Site: For more information, refer to the specificdocumentation on Cisco ACI Multi-Pod and Cisco ACI Multi-Site, including the respective releasenotes.For information about Cisco ACI Multi-Site hardware requirements, see the following tches/datacenter/aci/aci lti-Site-Hardware-Requirements-Guide-201.htmlFor more information about the differences between the Cisco Nexus 9500 platform module line cards, refer tothe following -c78732088.htmlThe Cisco ACI fabric forwards traffic based on host lookups (when doing routing): all known endpoints in thefabric are programmed in the spine switches. The endpoints saved in the leaf switch forwarding table are onlythose that are used by the leaf switch in question, thus preserving hardware resources at the leaf switch. As aconsequence, the overall scale of the fabric can be much higher than the individual scale of a single leaf switch.The spine switch models also differ in the number of endpoints that can be stored in the spine proxy table,which depends on the type and number of fabric modules installed.You should use the verified scalability limits for the latest Cisco ACI release and see how many endpoints canbe used per e.html#Verified Scalability GuidesAccording to the verified scalability limits, the following spine switch configurations have the indicated endpointscalabilities: Max. 450,000 Proxy Database Entries with four (4) fabric line cards Max. 180,000 Proxy Database Entries with the fixed spine switchesThe above numbers represent the sum of the number of MAC, IPv4, and IPv6 addresses; for instance, in thecase of a Cisco ACI fabric with fixed spine switches, this translates into: 180,000 MAC-only EPs (each EP with one MAC only) 90,000 IPv4 EPs (each EP with one MAC and one IPv4) 60,000 dual-stack EPs (each EP with one MAC, one IPv4, and one IPv6)The number of supported endpoints is a combination of the capacity of the hardware tables, what the softwareallows you to configure, and what has been tested.See the Verified Scalability Guide for a given release and to the Capacity Dashboard in the Cisco APIC GUI forthis information. 2022 Cisco and/or its affiliates. All rights reserved.Page 7 of 218

CablingDetailed guidelines about which type of transceivers and cables you should use is outside of the scope of thisdocument. The Transceiver Compatibility Matrix is a great tool to help with this task:https://tmgmatrix.cisco.com/Cisco Application Policy Infrastructure Controller (APIC)The Cisco APIC is the point of configuration for policies and the place where statistics are archived andprocessed to provide visibility, telemetry, and application health information and enable overall management ofthe fabric. The controller is a physical appliance based on a Cisco UCS rack server with two interfaces forconnectivity to the leaf switches. The Cisco APIC is also equipped with Gigabit Ethernet interfaces for out-ofband management.For more information about the Cisco APIC models, see the following 715.htmlNote:A cluster may contain a mix of different Cisco APIC models: however, the scalability will bethat of the least powerful cluster member. The naming of the Cisco APICs, such as M3 or L3, isindependent of the UCS series names.Fabric with mixed hardware or softwareFabric with different spine switch typesIn Cisco ACI, you can mix new and old generations of hardware for the spine and leaf switches. For instance,you could have first-generation hardware leaf switches and new-generation hardware spine switches, or viceversa. The main considerations with spine hardware are as follows: Uplink bandwidth between leaf and spine switches Scalability of the spine proxy table (which depends primarily on the type of fabric line card that is usedin the spine) Cisco ACI Multi-Site requires spine switches based on the Cisco Nexus 9500 platform cloud-scale linecards to connect to the intersite networkYou can mix spine switche switches of different types, but the total number of endpoints that the fabricsupports is the minimum common denominator.Fabric with different leaf switch typesWhen mixing leaf switches of different hardware types in the same fabric, you may have varying support offeatures and different levels of scalability.In Cisco ACI, the processing intelligence resides primarily on the leaf switches, so the choice of leaf switchhardware determines which features may be used (for example, multicast routing in the overlay, or FCoE). Notall leaf switches provide the same hardware capabilities to implement all features.As an example, classification features such as IP address-based EPG, copy service, service-based redirect,FCoE, and potentially microsegmentation (depending on whether or not you use a software switch that supportsthe OpFlex protocol) or Layer 3 multicast are not equally available on all leaf switches. 2022 Cisco and/or its affiliates. All rights reserved.Page 8 of 218

Cisco APIC pushes the managed object to the leaf switches regardless of the ASIC that is present. If a leafdoes not support a given feature, it raises a fault. For multicast routing you should ensure that the bridgedomains and Virtual Routing and Forwarding (VRF) instances configured with the feature are deployed only onthe leaf switches that support the feature.Fabric with different software versionsThe Cisco ACI fabric is designed to operate with the same software version on all the APICs and switches.During upgrades, there may be different versions of the OS running in the same fabric.If the leaf switches are running different software versions, the following behavior applies: Cisco APIC pushesfeatures based on what is implemented in its software version. If the leaf switch is running an older version ofsoftware and the Cisco APIC does not understand a feature, the Cisco APIC will reject the feature; however, theCisco APIC may not raise a fault.For more information about which configurations are allowed with a mixed OS version in the fabric, refer to thefollowing tml#Software and Firmware Installation and Upgrade GuidesRunning a Cisco ACI fabric with different software versions is meant to be just a temporary configuration tofacilitate upgrades, and minimal or no configuration changes should be performed while the fabric runs withmixed OS versions.Fabric extenders (FEX)You can connect fabric extenders (FEXes) to the Cisco ACI leaf switches; the main purpose of doing so shouldbe to simplify migration from an existing network with fabric extenders. If the main requirement for the use ofFEX is the Fast Ethernet port speeds, you may want to consider also the Cisco ACI leaf switch models CiscoNexus N9K-C9348GC-FXP, N9K-C93108TC-FX, N9K-C93108TC-FX-24, N9K-C93108TC-EX, N9KC93108TC-EX-24, and N9K-C93216TC-FX2.A FEX can be connected to Cisco ACI with what is known as a straight-through topology, and vPCs can beconfigured between hosts and the FEX, but not between the FEX and Cisco ACI leaf switches.A FEX can be connected to leaf switch front-panel ports as well as converted downlinks (since Cisco ACIrelease 3.1).A FEX has many limitations compared to attaching servers and network devices directly to a leaf switch. Themain limitations as follows: No support for L3Out on a FEX No Rate limiters support on a FEX No Traffic Storm Control on a FEX No Port Security support on a FEX A FEX should not be used to connect routers or Layer 4 to Layer 7 devices with service graph redirect The use in conjunction with microsegmentation works, but if microsegmentation is used, then Quality ofService (QoS) does not work on FEX ports because all microsegmented traffic is tagged with a specific 2022 Cisco and/or its affiliates. All rights reserved.Page 9 of 218

class of service. Microsegmentation and a FEX is a feature that at the time of this writing has not beenextensively validated.Support for FCoE on a FEX was added in Cisco ACI release /datacenter/aci/apic/sw/1-x/release/notes/apic rn 221.htmlWhen using Cisco ACI with a FEX, you want to verify the verified scalability limits; in particular, the limits relatedto the number of ports multiplied by the number of VLANs configured on the ports (commonly referred to as html#Verified Scalability GuidesWith regard to scalability, you should keep in mind the following points: The total scale for VRFs, bridge domains (BDs), endpoints, and so on is the same whether you areusing FEX attached to a leaf or whether you are connecting endpoints directly to a leaf. This meansthat, when using FEX, the amount of hardware resources that the leaf provides is divided among moreports than just the leaf ports. The total number of VLANs that can be used on each FEX port is limited by the maximum number ofP,V pairs that are available per leaf switch for host-facing ports on FEX. As of this writing, this numberis 10,000 per leaf switch, which means that, with 100 FEX ports, you can have a maximum of 100VLANs configured on each FEX port. At the time of this writing, the maximum number of encapsulations per FEX port is 20, which meansthat the maximum number of EPGs per FEX port is 20.The maximum number of FEX per leaf switch is 20. For more information about which leaf switch is compatible with which fabric extender, refer to the y/fexmatrix/fextables.htmlFor more information about how to connect a fabric extender to Cisco ACI, see the following c-Extender-with-Applica.htmlPhysical topologyAs of release 4.1, a Cisco ACI fabric can be built as a two-tier fabric or as a multi-tier (three-tiers) fabric.Prior to Cisco ACI 4.1, the Cisco ACI fabric allowed only the use of a two-tier (spine and leaf switch) topology,in which each leaf switch is connected to every spine switch in the network with no interconnection betweenleaf switches or spine switches.Starting from Cisco ACI 4.1, the Cisco ACI fabric allows also the use of two tiers of leaf switches, whichprovides the capability for vertical expansion of the Cisco ACI fabric. This is useful to migrate a traditionalthree-tier architecture of core-aggregation-access that have been a common design model for manyenterprise networks and is still required today. The primary reason for this is cable reach, where many hosts arelocated across floors or across buildings; however, due to the high pricing of fiber cables and the limitations ofcable distances, it is not ideal in some situations to build a full-mesh two-tier fabric. In those cases, it is more 2022 Cisco and/or its affiliates. All rights reserved.Page 10 of 218

efficient for customers to build a spine-leaf-leaf topology and continue to benefit from the automation andvisibility of Cisco ACI.Figure 2 Cisco ACI two-tier and Multi-tier topologyLeaf and spine switch functionsThe Cisco ACI fabric is based on a two-tier (spine and leaf switch) or three-tier (spine switch, tier-1 leaf switchand tier-2 leaf switch) architecture in which the leaf and spine switches provide the following functions: Leaf switches: These devices have ports connected to classic Ethernet devices, such as servers,firewalls, and router ports. Leaf switches are at the edge of the fabric and provide the VXLAN TunnelEndpoint (VTEP) function. In Cisco ACI terminology, the IP address that represents the leaf VTEP iscalled the Physical Tunnel Endpoint (PTEP). The leaf switches are responsible for routing or bridgingtenant packets and for applying network policies. Spine switches: These devices interconnect leaf switches. They can also be used to build a Cisco ACIMulti-Pod fabric by connecting a Cisco ACI pod to an IP network, or they can connect to a supportedWAN device (see more information in the "Designing external layer 3 connectivity" section). Spineswitches also store all the endpoints-to-VTEP mapping entries (spine switch proxies).Within a pod, all tier-1 leaf switches connect to all spine switches, and all spine switches connect to all tier-1leaf switches, but no direct connectivity is allowed between spine switches, between tier-1 leaf switches, orbetween tier-2 leaf switches. If you incorrectly cable spine switches to each other or leaf switches in the sametier to each other, the interfaces will be disabled. You may have topologies in which certain leaf switches arenot connected to all spine switches (such as in stretched fabric designs), but traffic forwarding may besuboptimal in this scenario.Leaf switch fabric linksUp until Cisco ACI 3.1, fabric ports on leaf switches were hard-coded as fabric (iVXLAN) ports and couldconnect only to spine switches. Starting with Cisco ACI 3.1, you can change the default configuration and makeports that would normally be fabric links, be downlinks, or vice-versa. More information can be found at thefollowing b ACIFundamentals/b ACI-Fundamentals chapter 010011.html#id 60593For information about the optics supported by Cisco ACI leaf and spine switches switches, use the followingtool: 2022 Cisco and/or its affiliates. All rights reserved.Page 11 of 218

https://tmgmatrix.cisco.com/homeMulti-tier design considerationsOnly Cisco Cloudscale switches are supported for multi-tier spine and leaf switches. Spine: EX/FX/C/GX spine switches (Cisco Nexus 9332C, 9364C, and 9500 with EX/FX/GX line cards) Tier-1 leaf: EX/FX/FX2/GX except Cisco Nexus 93180LC-EX Tier-2 leaf: EX/FX/FX2/GXDesign considerations for multi-tier topology include the following: All switch to switch links must be configured as fabric ports. For example, Tier-2 leaf switch fabricports are connected to tier-1 leaf switch fabric ports. A tier-2 leaf switch can connect to more than two tier-1 leaf switches, in comparison to a traditionaldouble-sided vPC design, which has only two upstream switches. The maximum number of ECMP linkssupported by a tier-2 leaf switch to tier-1 leaf switch is 18. An EPG, L3Out, Cisco APIC, or FEX can be connected to tier-1 leaf switches or to tier-2 leaf switches. Tier-1 leaf switches can have both hosts and tier-2 leaf switches connected on it. Changing from a tier-1 to a tier-2 leaf switch and back requires decomissioning and recommissioningthe switch. Multi-tier architectures are compatible with Cisco ACI Multi-Pod and Cisco ACI Multi-Site. Tier-2 leaf switches cannot be connected to remote leaf switches (tier-1 leaf switches). Scale: The maximum number of tier-1 leaf switches and tier-2 leaf switches combined is equal to themaximum number of leaf switches in the fabric (200 per pod; 500 per Cisco ACI Multi-Pod as of CiscoACI release 5.1).For more information about Cisco ACI multi-tier, see the following astructure/whitepaper-c11-742214.htmlPer leaf switch RBAC (Role Based Access Control)Up until Cisco ACI 5.0, an Cisco ACI fabric administrator could assign a tenant to a security domain to let usershave read/write privilege for a specific tenant assigned to that security domain, but that RBAC feature was notapplicable to specific leaf.Starting from Cisco ACI 5.0, a leaf switch can be assigned to a security domain so that only specific users canconfigure leaf switches assigned to that security domain and users in other security domains have no a

As a best practice, short-lived software releases should be upgraded to the next available long-lived release for stability and longer maintenance benefits. At the time of this writing, Cisco