To LACP or not to LACP (on a 5.1 vDS)

http://www.poppingclouds.com/2012/12/20/to-lacp-or-not-to-lacp-on-a-5-1-vds-2/

I have been recently mulling over the potential benefit of LACP in some of our environments. I want to discuss how LACP is implemented in vSphere, its limitations, and the potential benefits that I see in its use. I will also go over the process for enabling LACP from the vSphere side of things.

Beginning with vSphere 5.1, VMware supports Link Aggregation Control Protocol (LACP) on distributed switches (vDSs). LACP, as I am sure you are already aware, allows the bundling together of multiple physical links to form a single logical channel. The purpose here is to provide more efficient network redundancy and failover (as well as increased available bandwidth, which I will get to in a moment).

LACP works by simultaneously sending frames down each interface that has been enabled for LACP. If the device on the other end of the connection is configured for LACP, it will also start sending frames along those same links thereby enabling both systems to detect multiple connections between themselves and combine them into a single logical link.

So this all sounds great, the immediate question becomes, “Do I enable this in my vSphere 5.1 environment?” And as any architect would tell you (really it’s always the same answer for any design question) the answer is, “It depends.”

First we need to look closely at the limits of the current implementation of LACP in 5.1. According to the 5.1 networking guide published by VMware, these are the limitations as they exist today:

Limitations:

• LACP is only available on vSphere Distributed Switches.
  • This means you need Enterprise Plus (or better) licensing.
• LACP only works with IP Hash load balancing and Link Status Network failover detection.
• LACP is not compatible with iSCSI software multipathing.
• vSphere only supports one LACP group per distributed switch, and only one LACP group per host.
• LACP settings do not exist in host profiles.
• LACP between two nested ESXi hosts is not possible.
• LACP does not work with port mirroring.

Now, depending on your environment’s functional and non-functional requirements, one or more of these may be a show stopper. For instance, if you have a budgetary constraint preventing you from purchasing Enterprise Plus licensing, that would undoubtedly rule this out. Or perhaps you are (like many environments today) aggregating all of your traffic from the host up through a pair of 10G interfaces into a stacked switch pair on a single vDS. If you are using ISCSI storage with software-based multipathing, you are probably going to run into a complication here.

You also need to be aware that the IP Hash load balancing method is not going to pay attention to the NIC utilization. Imagine a situation where a VM is accessing a backup server, it is quite possible that that VM is already saturating that link. It will never choose another uplink as long as the IP hash for it’s destination remains the same.

Okay, but don’t I get more bandwidth with the aggregated links?

Well yes and no. If you look at things from the point of view of a VM sitting in a port group, using IP Hash load balancing will let that single source IP use a single physical interface to any given destination IP. So for that one connection, no, you only have the bandwidth that is available on that single physical interface. But if we take a step back and look at a VM that is connecting to “many” destination IP addresses (for instance a web server), then yes, as a whole, that VM will have access to the total aggregate bandwidth available in the LACP group (even though each individual connection only has the total amount of bandwidth present in a single uplink). Likewise, inbound traffic will be subject to the load balancing policy that is in place on the access layer switches.

You also need to take into account the additional overhead of IP Hash, and determine if it is worth it for  your use case. For instance, the VMkernel will need to select the appropriate uplink for each connection while using IP Hash. Now imagine you have a VM that is accessing a backup server or a backend database for 90%+ of its traffic. The IP Hash calculation is useless in that scenario since it will ALWAYS choose the same physical interface (unless the IP on the remote host changes and alters the hash value).  Yet the VMkernel will still be making that calculation for every connection from that VM, even though it is always going to result in the same hash.

Lets take a look at a logical diagram of this (click to enlarge):

As you can see here, the LACP “magic” occurs pretty far up in the stack. The port groups are still subjected to the load balancing algorithm, and will be assigned one of the available physical uplinks based on that algorithm (IP hash) for that individual connection.

Bottom line: LACP can give you some pretty nice benefits in regards to failover and network failure detection. However, (in my opinion) it is not going to be better at balancing traffic, or route traffic any more efficiently than a properly configured set of static uplinks using “Route based on physical NIC Load.” Please keep in mind that I am basing that off of my research here, not on any real-world testing of LACP vs Nic Load. However it would have to be my recommendation that when using a vDS, you should stick with load-based teaming (Route based on physical NIC Load) rather than IP Hash for most use cases. It is load-aware, less complex to set up, places less overhead on the VMkernel (load calculations are made only every 30 seconds), and will route traffic just as efficiently (in my opinion) as IP Hash.

So, all that aside, there are still some notable advantages to running LACP within your vSphere environment.

Advantages:

• It is literally “Plug and play.”
• Link failures are detected nearly instantaneously and failover is immediate.
• Can detect cabling mistakes and reconfigure the links automatically.

So, lets assume that you have made the decision to use LACP. You have confirmed that your upstream switches support the protocol, the limitations of LACP in a 5.1 VDS are not an issue for your environment, and you have a valid use case that warrants it. How do you go about enabling it?

It’s actually fairly simple. I will go through the steps below.

How to configure LACP on a vSphere 5.1 vDS:

First thing is first, you must use the vCenter Web client for this.

In order to use the dynamic link aggregation configuration, you must be working on a v5.1.0 switch or later:

It is also a good idea to enable the LLDP (Link Layer Discovery Protocol) since this makes the configuration on the physical switch a bit simpler:

With LLDP enabled, it is very easy to confirm which exact ports on the physical switch is connected to which individual hosts in the cluster.

Now go over to the uplink port group on your VDS. You will see the new LACP option:

You will notice, that you can also set the mode to either active or passive. This is the mode in which the vDS decides to initiate the LACP negotiation. In Passive mode, it will remain silent and not transmit any LACP BPDU frames unless the switch on the other side initiates the session. In active mode, the vDS will start transmitting the frames to start the session with the switch. Passive is the default option.

The last thing we need to do from a vDS standpoint is configure all the other port groups on this switch to use the “Route based on IP hash” load balancing method. Note that you also need to make sure that “Network failure detection” is set to “Link Status only.” Beacon probing is not supported with IP Hash load balancing. Also make sure all uplinks are active. Standby and unused uplinks are also not supported with IP Hash load balancing.

From this point you simply need to configure the LACP port grouping on the physical switch, (or switches if stacked). Again, having LLDP enabled makes this task a bit easier, especially if we are dealing with a ton of switchports here.