A bit of background
I've been lucky enough recently to work with FCoE a fair bit, using IBM Systems Networking switches both rack-mount and integrated into the IBM Flex/PureFlex chassis. What I've found is that getting FCoE running with IBM equipment is actually really easy. I thought I'd try and condense the wealth of information in the Application Guides down to a blog post.
Through talking with people about FCoE, and especially the IBM equipment, I get the impression that FCoE is a fairly common method these days for implementing storage fabric, however not everyone I speak with has an understanding of how it all hangs together, so I'll attempt to also cover the general theory behind FCoE where necessary.
I think everybody has at least heard of FCoE by now. If you are fairly unfamiliar with it, FCoE is basically a way to carry Fibre-Channel storage traffic using Ethernet switching equipment. The equipment needs to support FCoE, namely CEE/DCB for transmitting frames, and FIP snooping (or FIPS) for inspecting FCoE FIP packet contents to ensure packets are only transmitted between valid FCoE endpoints.
You also require an FCF (Fibre Channel Forwarder) somewhere in your FCoE fabric (storage network), which dissects the FCoE packets for the purposes of WWN zoning or NPIV (N-Port Identification Virtualisation). The FCF is then connected to your storage via plain old FC, or FCoE, completing the fabric.
Let's consider an example scenario, with an IBM EN4093 Flex chassis switch, connected to an 8264CS converged fabric switch/FCF. We'll assume the Omni (switchable Ethernet/FC SFP+ ports) on the 8264CS switch are connected to existing Fibre-Channel storage.
All of the below commands require your log on to the switch CLI via SSH, Telnet or using a serial cable. Setting up these connections is covered in the installation documentation for the switches.
tagging, switchport, ISCLI?
You may notices that subtle differences (and some overt ones) exist between the EN4094 and 8264CS configuration syntax on the switches you are trying this on. The CLI used on IBM switches is in the process of transitioning between IBMNOS (the old BNT style configs) and ISCLI - the new "industry standard" (read: Cisco) style configuration. The syntax changes slightly each firmware release to make it more intuitive, and more familiar to people without IBM/BNT switch experience. Check the firmware versions of your switches if any of the below does not work, I recommend updating to at least version 7.7. The updates can be downloaded from IBM FixCentral.
On the EN4094
Start by enabling CEE on the Chassis switch. This will allow DCB/CEE capable servers connected to the switch to negotiate via DCBX to use DCB/CEE instead of standard Ethernet.
You will then need to create a VLAN for FCoE traffic. I suggest using VLAN 1002 unless you have a reason to change it, as most FCoE HBAs will default to VLAN 1002, and you might save yourself some configuration on the server's HBA and OS.
#enter the vlan configuration context for VLAN 1002 vlan 1002 #Just a descriptive name to make the config easier to read, not required name "FCoE" #An example of adding a single physical port to the FCoE network. #Not required if you are only using UFP or vNICs to connect your servers. member INTA1 #Replace with the ports your 8264CS switch are connected to (uplinks) member EXT13,EXT14 #An example of adding a virtual NIC (UFP or vNIC) to the FCoE network. #Not required if you are not using vNICs/UFP vmember INTA2.2 #exit VLAN configuration context exit
Enable tagging on your uplink ports, as you will likely want to carry other VLANs. It is also possible to have dedicated uplinks for FCoE traffic. If you are doing that - you can skip this step.
#Enter the configuration context for the two #external uplink ports. Substitute EXT13,EXT14 #for your uplink ports. interface EXT13,EXT14 #A descriptive name for the ports, again, not required. name "Uplink" #Enable tagging (trunking in Cisco-speak) tagging #Exit the configuration context exit
Then enable FIP Snooping, so that FCoE sessions can be detected and established on the switch.
fcoe fips enable`
This is all that is required on the access (chassis) switch to pass FCoE traffic through to the uplink ports.
On the 8264CS
The 8264CS is a fairly new switch in the IBM lineup. It is basically an 8264 top-of-rack switch, with an additional full FC switch, and can act as a Fibre Channel Forwarder. It can therefore be directly attached to FC devices, the zoning being done on the switch, but also supports NPV/NPIV. I'll cover both NPIV and FCF modes of operation.
The Familiar Stuff
We will need to configure CEE, FIPS and VLANs. We covered these when we configured the EN4093, so I'll write these without much explanation.
#Enable CEE cee enable #Configure FCoE VLAN vlan 1002 name "FCoE" #the ports connected to your access switch, in #this example, the EN4093. member 20,21 exit #Configure links to access switch (EN4093) interface 20,21 name "EN4094 Chassis Switch" tagging exit #Enable FIPS fcoe fips enable
Omni/FC Port Configuration
The first thing we have to do before we can use the 8264CS to complete the FCoE fabric, is to enable some FC ports. Even if you are connecting to FCoE-enabled storage using RJ45 copper links, you need to have some Omni ports configured as FC ports. This seems like an odd limitation, until you consider that there are separate Ethernet and FibreChannel boards in the 8264CS. Configuring some FC ports allows the FC switch board to participate in the FCoE VLAN, and provide FC switching/zoning or NPIV to WWNs in the FCoE fabric.
If you do not have any native FC devices/switches that you would like to connect your FCoE fabric to, just add some dummy FC ports. Ports need to be added in top/bottom pairs, so instead of just adding port 53, you will need to add 53 and 54.
#Configure Omni ports 53 and 54 as FC ports system port 53,54 type fc #Add the FC ports into your FCoE VLAN, so that #the built-in FC switch can dissect and vlan 1002 member 53,54 exit
NPIV is a great way to avoid compatibility issues that often spring up when trying to join FC fabrics with switching equipment from different vendors, or for avoiding fabric domain limits in larger fabrics. In NPIV, each endpoint has it's own virtual WWN, that appears as a Virtual N-Ports in the upstream FC fabric. To enable this, you have to configure your FCF to provide NPIV on your FCoE VLAN, and specify the ports that connect to the FC fabric in the NPIV traffic map.
NPIV, like FCF mode, is enabled on a per-VLAN basis.
#Enter the FCoE VLAN configuration context vlan 1002 #Enable NPV, which provides NPIV - confusing much? npv enable #Configure FC ports connected to fabric as #external ports in the traffic map. These will #be your dummy FC ports if you are using only FCoE #and no FC connected devices! npv traffic-map external-interface 53,54 #exit VLAN context exit
You will then need to configure zoning. This will either be done on the storage itself. In the case of the v3700/v7000, you will want to create hosts for each virtual N port. The WWNs you can use for zoning are shown on the 8264CS by running the below command.
This will show all FCoE connections. Note: WWNs will only be visible on your FCF, which is the 8264CS in this case.
show fcoe fips fcoe
The 8264CS is a capable FC switch, and can provide an FC fabric of it's own. It can also be joined to an existing fabric. It is good standard advice when connecting FC switches from different vendors to look out for compatibility issues. Don't assume that switches from different vendors are correctly passing traffic through the fabric, test thoroughly. I have personally tested IBM switches with Brocade switches and had no issue, but if you do have issues, consider using NPIV between two separate fabrics, one fabric for each vendor.
To FCF mode on the switch, you have to enable FCF on your FCoE VLAN.
#Our FCoE VLAN, which are very familiar with by now. vlan 1002 #Enable FCF for traffic in this VLAN fcf enable #Exit configuration context exit
The next step, is to create your zone. You will need the WWNs of your HBAs, and the WWNs of your storage controllers. Create separate zones if your servers have multiple paths to the storage controllers, for example if they have multiple HBAs you would like to connect to the same LUNs, in order to provide multi-pathing.
#First, create aliases for your WWNs. Not required, #but much easier in the long-run if you need to change #WWNs on a server or storage controller! #Alias for your server's HBA. Replaces the x's with #the actual WWN, be careful to include the colons between segments. fcalias ServerHBA wwn xx:xx:xx:xx:xx:xx:xx:xx #Alias for your storage controller, you will likely have multiple #controllers for the same storage, so create more aliases, and #include them in the zone if required. fcalias StorageController xx:xx:xx:xx:xx:xx:xx:xx #Create a zone named FirstZone zone name FirstZone #Add aliases of HBA, and Controller. member fcalias ServerHBA member fcalias StorageController #exit this context exit #Now create a zone set. We only have one zone, but #if you have multiple related zones, you can group #them logically. zoneset name ActiveConfig member FirstZone exit #Now make the zone set, and zone, active. zoneset activate name ActiveConfig
You should then be able to rescan your HBAs and see the storage controllers on the server.
Easy, right? Feel free to send me any questions, or let me know if you hit any snags. I hope this is useful, and that it saves you some time. Thanks for reading.