FEX

Understanding Cisco NEXUS 2000 Series Fabric Extender (FEX)

1. N2K acts as a remote line card for Cisco Nexus switches with no local switching. N2K is managed by a Parent switch such as 5K, 6K, 7K or 9K.

2. N2k can’t switch traffic locally. All traffic is sent to the parent switch that provides central forwarding and policy enforcement, even including host communication between two hosts, connected to the same FEX.

3. N2K Terminology:

1. Fabric interface: fabric interface is also known as Network interface. Ethernet uplink port designated for connection from FEX to its parent switch.Support static port-channel.

2. Host Interface: An Ethernet interface which is used to connect host or server. FEX host interface are edge port (Portfast enabled) as well as BPDU guard enabled. If we connect switch to host interface that interface is placed in “err-disabled” state when BPDUs are received. We can’t disable it. Or we can say host interface can’t be changed to “spanning-tree port type network/normal”. BPDU guard is enabled by default and can’t be disabled.

4. N2k Management:

All management is performed from parent switch. There is no console and no vty on FEX. NX-OS is automatically downloaded from parent switch. Parent switch discovers the FEX by detecting the fabric interface of FEX. Parent switch and FEX establishes in-band IP connectivity with each other. The Parent switch assigns an IP Address in the range of Loopback addresses to FEX to avoid potential conflicts with IP addresses that might be in use on the network. FEX doesn’t store any configuration locally.

5. Forwarding:

As we already discussed N2K doesn’t have local switching capabilities. All traffic is sent to the parent switch that provides central forwarding and policy enforcement, even including host communication between two hosts, connected to same FEX. We have a concept called VN Tag which provide the capability to differentiate traffic between different host interfaces traversing the fabric uplinks.

1. When frame received on N2K Host interface then N2K adds a unique VN-Tag and forward frame to fabric uplinks.
2. Parent switch do switching here means do lookup and forward the frame.
3. N2K receive the frame with specific VN-Tag and strip the VN-Tag and frame is forwarded to host interface.

The VN-Tag header consists of the following fields:

6. There are two methods of configuring/connecting fabric interfaces:

1. Static pinning Fabric interface connection:The static pinning feature allows groups of access interface (host interface) to be bound to particular fabric interface. If one particular uplink fails, a range of FEX ports associated with this uplink will also fail.

Changing the values of max link command is disruptive, all the host interfaces on FEX are brought down and back up as parent switch reassigns its static pinning.

2. Etherchannel Fabric Interface Connection: when we configure fabric extender to use an etherchannel fabric interface connection to its parent switch. A fabric interface that fails in the etherchannel will not trigger a change to a host interface. Traffic is automatically redistributed across remaining links in the etherchannel fabric interface.

7. Installation of FEX:
Assuming FEX feature is enabled at Parent switch if not then first we have to enable FEX feature.

Create a FEX using below command:
Eva-parent-sw1(config)# Fex 101
(100-199/this is considered as module number)
Eva-parent-sw1(config-fex)# pinning max-links 1
Change in Max-links will cause traffic disruption.
Eva-parent-sw1(config-fex)# description “eva-fex21”

Associate FEX to the interface (port-channel or physical)

Eva-parent-sw1(config)# interface port-channel101
Eva-parent-sw1(config-if)# description “eva-fex21”
Eva-parent-sw1(config-if)# switchport mode fex-fabric
Eva-parent-sw1(config-if)# fex associate 101
Eva-parent-sw1 (config-if)# no shut

We have to add physical interface to port-channel using below command:

Eva-parent-sw1(config)# interface Ethernet X/Y
Eva-parent-sw1(config-if)# description “eva-fex21 Uplink X”
Eva-parent-sw1(config-if)# switchport mode fex-fabric
Eva-parent-sw1(config-if)# fex associate 101
Eva-parent-sw1(config-if)# channel-group 101
Eva-parent-sw1(config-if)# no shut

Always check FEX number, port-channel number and interface which we are going to use are not already in use before starting the change.

Show fex
Show fex detail
Show interface status fex 101
Show interface fex-fabric
show fex XXX transceiver
show int eth1/1 capabilities
show int eth1/1 transceiver
sh int status | in 1/28
debug fex pkt-trace

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++