NIC Teaming is reducing throughput. Why?

Hello everyone!

I have been trying to improve our network here. First we acquired two very nice HP 5900 10 Gbps switches (48 port). Got a bunch of Intel 10Gb NICs. The plan is to have redundant network connectivity and maximize network throughput. I am having a problem... 

Short version:

Using iPerf I can saturate 10Gb links easily (meaning I get nearly 10Gbps) however when I team the same ports, I get only 3-4 Gbps. Anything I could do to improve the network throughput over a teamed link?

Long version:

Here is how two physical test machines are connected, network wise:

  • The two network switches are configured to behave as a single chassis (using HP IRF).
  • Each machine has a dual port 10GB NIC. Top port goes to switch 1. Bottom port goes to switch 2.
  • All connections are via CAT 7 patch cords (really nice ones).
  • IP of team on server 1 : 192.168.3.30
  • IP of team on server 2: 192.168.3.13

Test results:

I am using iPerf for running the throughput tests. This tool eliminates any disk I/O type bottlenecks on the machines.

> Iperf is in server mode on server 2

> Iperf is in client mode on server 1

> TCP window size has been maximized to 1.5g

> Jumbo Frames have been setup properly and tested to be working fine on all ports.

Notice that I get only around 3.5 Gbps:

Teaming settings:

Here is what I have set it up like (for now). I can setup LACP later.

I do want to double the throughput "eventually" but for now I expect at least the same performance as a single link.

Any tips?

March 24th, 2015 11:47am

Hi Rajeev,

If we choose the Swith Independent mode, the inbound traffic will only received by single NIC.

Dynamic mode may cause out-of-order delivery of packets.

Please try to change the Load balancing mode to Address hash distribution.

Besides, teaming doesn't increase bandwidth to a single user.  Teaming increases the aggregate throughput, which means that more people can use the speed of the individual NICs simultaneously.

For detailed information about NIC teaming, please click the link below:

https://www.microsoft.com/en-us/download/details.aspx?id=40319

Best Regards

Free Windows Admin Tool Kit Click here and download it now
March 24th, 2015 11:34pm

Oh wow! the moment I changed both servers to Switch Independent / Address Hash / all adapters active, I am (almost) able to saturate the 10Gb connection:

So thanks a ton! :-)

Now lets see if I can increase the streams in iPerf and see higher than 10Gbps....

Will LACP give me higher throughput over a single stream?

March 25th, 2015 1:31pm

I tried iPerf with 2 and 4 simultaneous streams over the teamed NICs. (see screenshots below)

Two streams topped off around 8-9 Gbps total

Four streams varied around 8 to 18 Gbps.

So, overall the throughput is rather flaky in multistream mode. It is not 20Gbps, and that is probably just wishful thinking. Well, as long as it is at or above a single link's speed -10Gbps, I am "almost" happy :-)

Will try LACP next.....I guess this will take longer since switch needs to be configured.

Free Windows Admin Tool Kit Click here and download it now
March 25th, 2015 2:15pm

Hi Rajeev,

Glade to hear that your issue is "almost" resovled and thanks for the sharing.

Actually, from my point of view, network speed depends many factors. Normally, we won't and can't let it reach the limit. The 20Gbps is a theoretical value. Teaming will cause additional cost.

>>Will LACP give me higher throughput over a single stream?

As I have mentioned above, theoretically, it can let single stream distributed on multiple NICs.(According to the loadbalance setting of the switch) But it will cause out-of-order delivery of packets. It will cause additional cost to reorder the packets.

Teaming doesn't increase bandwidth to a single user.  Teaming increases the aggregate throughput, which means that more people can use the speed of the individual NICs simultaneously

Best Regards.

March 25th, 2015 11:27pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics