Multichannel (UFP and vNIC) on IBM Switches - Sat, Sep 13, 2014

Hardware density has become increasingly important over the last several years. Being able to do more compute with a smaller physical and energy footprint has become the new focus for IT. In order to do this effectively, more and more businesses are moving traditional workloads to the cloud, whether it be public, private or hybrid.

We often find however that all of this consolidation requires previously separated network and storage fabrics to coexist on the same physical hardware, virtualised, especially in private cloud scenarios. Traditionally that has meant introducing more and more IO adapters into systems to cope with the number of isolated fabrics, but we are trying to make everything smaller and cheaper; This is where converged network adapters and multi-channel networking becomes relevant.

Network adapters such as the Emulex and Brocade CNAs are physical network adaptors which have the ability to provide multiple virtual network or storage adaptors in a single card. Multiple communications channels are established between the CNA (installed in a server) and a compliant network switch. When working with IBM switches, these virtual NICs or HBAs (channels) are configured depending on the multi-channel mode set in the server's UEFI menu.

There are several ways in which multi-channel can be configured between IBM switches and supported CNAs. The most common configuration modes are Switch-Independent vNIC, vNIC and UFP. These modes are mostly the same in functionality, with the exception of UFP which adds some additional functionality. The main way in which these modes differ, is how and where the channels are configured. Let's step through the individual modes and how they are configured.

Multi-Channel operation is configured by first entering the adaptor configuration menu, under the server's UEFI menu. On IBM servers, you can enter UEFI by pressing F1 when prompted during server boot up. It may help to take node of the MAC address and PCI address of the adaptor you would like to configure, prior to entering the setup menu, if you have an OS installed on the server. Once in the UEFI menu, you navigate to System Settings -> Network, and select the device you would like to configure multi-channel mode on. For Emulex adaptors, you can enter the Emulex NIC Setting screen the change the multi-channel mode. Once in this screen, you can select supported Multi-Channel modes by altering the Multichannel setting.

For UFP, you select UFP as the multi-channel mode. With UFP, the remaining configuration is done in the configuration of a supported switch. Negotiation between the NIC in the server and the switch to establish UFP channels and settings normally only happens at server boot, so remember that once the mode is changed in the UEFI menu, and when any changes are made to the UFP configuration on the switch, the server will need a reboot to pick up the changes.

An example switch configuration section is below, which we will walk through.

#configuration for the first vNIC on physical port 10
ufp port INTA1 vport 1
network mode trunk #Configure this port as a trunk port
network default-vlan 1 #Not required when using VLAN 1, just shown for example purposes. Similar to setting native-vlan on physical ports.
qos bandwidth min 40 #Set the QoS to allocate 40% of bandwidth minimum. Useful for identifying which vNIC corresponds to a network interface in the OS, as this will change the advertised link speed show in the OS.
qos bandwidth max 40
enable
exit
!

#configuration for the second vNIC on physical port 10, configured for FCoE
ufp port INTA1 vport 2
network mode fcoe #Set the port mode to FCoE
network default-vlan 1003 #On IBM switches, the FCoE VLAN can be changed.
qos bandwidth min 60 #Set the QoS to allocate 60% of bandwidth minimum
qos bandwidth max 60 #Set the QoS to allocate no more than 60% of bandwidth
enable
exit
!

#Enable UFP on the port
ufp port INTA1 enable
!

#Enable UFP functionality globally
ufp enable
!

#Example VLAN to demonstrate different configuration for adding vNICs/UFP ports to a VLAN
vlan 100
#vmember is used to add virtual ports to a VLAN.
#Syntax is:
vmember INTA1.1
enable
!

#FCoE vlan example
vlan 1002
vmember INTA1.2
enable
!

As you can see from the configuration, virtual NICs or "UFP Channels" and their usage are configured on the switch side. All that needs to be done from the server side is to enable UFP as the multi-channel mode in the UEFI menu for the CNA.

Switch independent vNIC mode is different to UFP mode, in that the CNA allocates vNICs for each VLAN permitted on the switch port. As many switches are capable of using VLANs, even older ones which are not UFPcapable, this mode is considered switch-independent as the CNA is doing most of the heavy lifting when it comes to detecting and configuring vNICs. The majority of configuration is handled by the CNA. This mode can be selected in the UEFI menu, an example switch config for an EN4093 below could be used to allocate vNIC ports based on VLAN.

#This is our port connected to a CNA adapter configured for switch-independent vNIC mode
interface port INTA1
#Tagging is required on the port, as the CNA inspects tagged packets to map VLANs to vNICs
tagging
exit
!

#Our data VLAN, 1000
vlan 1000
enable
name "Data"
#Add the port as a member of this VLAN, allowing VLAN 1000 traffic to flow on INTA1. Note below that this port is also a member of VLAN 1002, our FCoE VLAN
member INTA1
!

#Our FCoE VLAN, 1004
vlan 1004
enable
name "FCoE"
#Add the port to this VLAN too, which in turn will cause the CNA to allocate another vNIC.
member INTA1
#NPV not required, just here for demonstration purposes
npv enable
npv traffic-map external-interface EXT11,EXT12
!

The above switch configuration does not carry any vNIC-specific configuration as you can see. The CNA does all the work in finding out how many vNICs need to be mapped to VLANs. Keep in mind, that your second vNIC will typically be used for any FCoE traffic.

Standard vNIC predates UFP, and typically if supported by your CNAs and switches you should go with UFP, as it is more flexible. Configuration-wise, it is very similar to UFP, except some of the configuration is performed in the UEFI menu in addition to the switch configuration. It is essentially in-between UFP and switch-independent vNIC when it comes to the CNA and switch's responsibility for configuration of the fabric. If you are on older switching or NIC/HBA hardware, or older firmware levels you might need to use this mode for compatibly.

To configure this mode, enter the UEFI menu on your server and enter the configuration for your virtual fabric adapter. Depending on your adaptor, you will be able to see the virtual fabric channels listed and you will be able to change the modes of each channel. This information will propagate to the switching hardware connected to the VFA on next reboot.

Switch configuration can be then used to configure VLAN membership on the virtual ports, which are referred to by their physical port number and channel. Example: INT4.1 Also, when adding vNIC ports to a VLAN, you normally need to use the vmember command instead of the regular member command.

An example for enabling VNIC and the first channel on physical port 4, allocating 50% of the physical bandwidth the port:

#Enable vNic
vnic enable
#Switch config context to vNic port 4.1
vnic port 4 index 1
#Allocate 50% bandwidth
bandwidth 50
enable
#Add another channel on the same port
vnic port 4 index 2
bandwidth 50
enable
#Add first channel on port 4 to VLAN 20
vlan 20
vmember int4.1

Keep in mind, your bandwidth percentages will need to add up to 100. Unlikely UFP, you need to set the modes of the various channels in UEFI if you need to use things like FCoE.

For consistency and stability, when FCoE is in use with multi-channel, ensure you use the second channel for FCoE traffic, unless you have reason to change it. The reason for this is that some CNAs, especially when using less sophisticated multi-channel modes (Switch-Independent vNIC for example) will use the second channel for FCoE by default, and it is non-trivial or not possible to change it, which can lead to some confusion if you are trying to configured a different channel on your switching equipment for FCoE use.