Proper NIC Teaming and Port Aggregation Configuration
Servers from HP, Dell, IBM and many other big name manufacturers, come with at least two or more Network Interface Cards built-in. Lately though, I have come to realize most network administrators do not know how to benefit from duel NIC’s. Frequently, one of the NIC’s are simply disabled or not cabled up. Occasionally, I will see a good effort made to make the most of it, but that’s pretty rare.
Let’s create a scenario. Pretend we have a server with two NIC’s. That server is plugged into a 10/100 manageable switch. Obviously, plugging one NIC in will allow for a full 100megs of bandwidth. But if we team the NICs and bond two switch ports together, then we can create a full 200meg pipe between the server and the switch. Performing these steps on multiple servers, means that you now have 200megs between each server. This is especially useful for backups which run over the network. The clients aren’t affected that much, because they are still at 100meg.
Link Aggregation Control Protocol, or LACP, is a protocol run on both the server and the switch. LACP is a negotiation protocol. This means that each device, the server and the switch, speak LACP to each other. Once they agree upon certain terms, the trunk is formed. LACP has two modes: Active and passive. If the device is in active mode, then it proactively seeks an LACP partner in which to form a trunk with. If in passive mode, the device will accept a request for trunking from another LACP host, but will not actively seek other LACP devices. This is important for one simple reason: Two devices in passive mode will never form a trunk. At least one must be in active mode. Or they both can be in active mode, but they both cannot be passive.
NIC Teaming
NIC Teaming is the process by which we create one logical NIC using both physical NIC’s. When the switch sends a frame down the wire to the server, it sees only one NIC, with one MAC address. How? When the NICs are teamed, both physical adapters have TCP/IP disabled on them. The server creates a logical (read: software based) NIC which TCP/IP is enabled and configured on.
Benefits of teaming are simple:
- Fault Tolerance. If one switch port, or one NIC goes down, the other remains online and passes traffic as if nothing happened at all.
- Load Balancing: The virtual NIC can distribute traffic evenly across the physical NICs. If your switch is 10/100, and both switch and server are set to 100meg, then you have a 200meg link.
- Increased Bandwidth. Do you use a disk based backup system to backup your server’s data over night? Teaming can decrease the time it takes to backup by increasing the bandwidth between the servers.
Server Configuration:
Configuration of the team is not difficult. It will however, blip the network connection to your servers. So it’s best to implement this during a scheduled outage.
Each server will usually come with a small application for managing the network interfaces. Each servers software is different based on the manufacturer of the NIC. So it’s up to you to locate this software. If you don’t see it, go to your server manufacturers web page and download the driver for the NIC. It should contain the software you need.
It’s difficult to walk you through the next step, as each NIC/Server is different. The utilities are generally very small and easy to use. Select both NICs and create the team. A wizard will usually ask you if you want fault tolerance, load balancing, or a plethora of other options. For the sake of this article, select and use LACP (Link Aggregation Control Protocol)
Once the wizard is complete, both NICs will be taken offline while the team is created. A few seconds later, the team is functioning. A reboot is usually not required.
Switch Configuration:
Many network equipment vendors sell switches capable of port aggregation. Port aggregation is to switches, what teaming is for NICs. Once again, we will be binding multiple interfaces together, to form one logical interface. Cisco calls it an “Etherchannel”. HP calls it “Tunking”. I’m sure other manufactures call it something different. Example configurations for both HP and Cisco follow:
Cisco
Switch# configure terminal Switch(config)# interface range fastethernet0/1 - 2 Switch(config-if-range)# switchport mode access Switch(config-if-range)# speed 100 Switch(config-if-range)# duplex full Switch(config-if-range)# channel-group 1 mode active Switch(config-if-range)# end
Briefly, using the “range” command allows us to specify more than one port at a time. Once in port configuration mode, we ensure the port is an access port. If necessary, make this a trunk port using the “switchport mode trunk” command. Speed and duplex are both hard coded to 100/full. I advise doing it this way as auto negotiation occasionally fails.
The “channel-group 1 mode active ” command, when run for the first time, will automatically create the logical channel group interface. “Mode active” enables LACP in active mode.
HP
ProCurve(config)# int c4-c6 lacp active
Simpler than Cisco’s configuration, and less feature-full, is HP’s one line trunk configuration. Used from global configuration mode, this command will enable LACP in active mode on ports c4-c6.
Things to keep in mind:
- Speed, duplex and flow control needs to be set the same way on all ports and on all devices. Any deviation could cause the trunk to not form. Because I’ve seen speed and duplex fail randomly, I recommend the ports be hard coded to the fastest speed allowed.
- Use descriptions in your switch configuration. This will help you better determine what the trunk/etherchannel is used for when your troubleshooting.
- NICs on the server must be identical.
- You don’t have to only trunk between a server and a switch. You can also do this on two switches, creating a 200meg uplink. Or as some would call it the “poor man’s fiber”
Teaming your server’s NICs has a whole host of benefits. Many of which could be felt as early as the next morning, when you check the duration of the backups and noticed they took half as long as they used to.
References:
