Hi,

I am trying to test the maximum throughput of my network adapters and switches, I have two 10Gb 570SFP+ Adapters in one server, and two of the same adapters in the other server. I have tried using the tool NTttcp using these commands

ntttcp.exe -s -m 8,*,10.10.1.2 -l 2M -a 2 -t 30

ntttcp.exe -s -m 8,*,10.10.5.2 -l 2M -a 2 -t 30

and for the receiver

ntttcp.exe -r -m 8,*,10.10.1.2 -rb 2M -a 16 -t 30

ntttcp.exe -r -m 8,*,10.10.5.2 -rb 2M -a 16 -t 30

I have a stacked HP 2920 switches (2 in a stack one side, and 2 in another stack the other), the switches are linked together via two fibre optics. I have already tested all adapters on an individual basis and I can get 10Gbps from one side to the other. However the switches are configured in such a way which are supposed to allow me to combine those two 10Gbps fibres and get me 20Gbps, however when I run the NTttcp tool for both adapters at the same time, i only get a maximum throughput of 10Gbps between the two adapters, not 10Gbps on each adapter. HP seem to think it's because the sending server can not send more than 10Gbps of data, however if i ran each of those commands at the same time in their own separate command window, i thought each command would max out each adapter. HP are asking if there is any way to increase the server's maximum sending throughput, i have already ran those commands with 2M, 20M and 50M but it doesn't make any difference.

The image attached shows how the switches and network are connected. the thin red lines are the 10Gbps adapters to the servers, the thick red lines are the fibre optics which link the two buildings (these are OM4 cables, which provide 10Gbps each)

Many thanks

Steve

Hi,

According to your description, my understanding is that you want to increase the servers send out throughout, in order to test the maximum throughput of the network adapter and switches.

Ntttcp.exe in multithreaded operation with one sender and two receivers(10.10.1.2 and 10.10.5.2). Try to add the -fr (full receive) option on the receiver and check to see if it works.

Besides, ntttcp.exe is a testing tool for network performance, the throughput of a system may be affected by various factors, including the limitations of underlying physical medium(network adapter, CPU, memory), drivers, and end client behavior.

Best Regards,
Eve Wang 

Hi,

I dont see a -fr option and when i try to use that it doesnt work. the CPU are 2 x 12 core intel xeons, and 128GB RAM, the drivers for each network adapter are the latest versions. each adapter can go at 10Gbps on its own, its just when i try to run the test on both NICS at the same time the throughput does down to roughly balance out at 5Gbps on each NIC

thanks

Steve

Hi,

In order to narrow down the problem, I suggest you to connect these 2 servers directly with network lines and simple switch. Use  the same command line to test the throughout, and check to see if the throughput may be increased.

Besides, have you tried any other test tools except for ntttcp.exe?  and what is the result

Best Regards,           
Eve Wang

unfortunately I can't test them connected directly because the one server is in another building, we link them fibre optic. i have tested one adapter at a time, and removing one of the fibres to ensure there is no fault and the throughput is 10Gbps using either of the fibres so i know each possible route is going at 10Gbps. i suspect it is a configuration issue with the switch but this is why i am trying to make sure that my sending tool is sending out at the max.

using the commands i mentioned previously and the NTttcp tool, should this be sending data to max out the NICs or not?

i used iperf as another test but i was only able to get 4Gbps out of that even using the same 2M to 5M parameters.

Hi,

The commands you mentioned with the NTttcp tool are no problem.

We usually use NIC teaming(also known as Load Balancing/Failover (LBFO), allows multiple network adapters to be placed into a team) in order to achieve the purpose of bandwidth aggregation with multi NICs. If both of your servers are Windows Server 2012/2012 R2, you may:
1. Open Server Manager, and click Local Server.
2. In PROPERTIES, click NIC Teaming.
3. Team these 2 NICs.

Detailed information about settings and introduction of NIC teaming you may reference the link below:
NIC Teaming Overview
http://technet.microsoft.com/en-us/library/hh831648.aspx

If earlier than Windows Server 2012, Windows Server system itself doesnt provide this function, different type of NIC may has its own setting method, about how to team multi NICs, you may contact the NICs vender for further support.

The content described above, is one of the possible reasons. It is better to isolate the intermediate devices, and confirm that there is no problem with the deployment of intermediate devices.

Best Regards,           
Eve Wang

hi,

these particular adapters are used for ISCSI and it is advised from official Microsoft web pages not to use NIC teaming and allow MPIO to handle the balancing.... will my NTttcp tests still allow for maxing out each adapter when configured in this way? i would imagine MPIO is only balancing out ISCSI traffic not the NTttcp. I could try the teaming to see if it makes a difference though. I suspect it is something on the switch which is configured wrong, I want to be sure that what i am running should be maxing out the NICs in the server, and from what you said above my commands shouldn't be a problem with this.

thanks

Steve

Just to update and close this thread - the bottleneck were the switches, i changed the load balancing policy from L3 to L4 and i managed to surpass the 10Gbps limit, albeit only to 12.2Gbps but that's another issue. the point is the command i was using did seem to be maxing out the NIC. thanks for all the help

Steve

Link aggregation assigns each frame on a frame-by-frame basis to one member of the aggregated link based on a hash of a property taken from that frame's headers or packet headers, eg, (dest.IP.Address:Source.IP.Address).

If the traffic you are generating is identical as to whichever property the hashing algorithm is using, then every frame will be assigned to the same link, thus "identical" traffic can never exceed the speed of any one member of the aggregated link.

Link aggregation is useful when you have multiple devices communicating with multiple devices, eg, on a distribution switch.

Link aggregation is *not* load balancing.

hi,

thanks for your additional comments on this, I have since spoke with HP and they made a change to a slightly different load balancing level and I exceeded the 10Gbps threshold, however it still doesn't seem to go above about 12Gbps. HP seem to think this is normal, that most people never get the 20Gbps - if that's true then I really don't understand the "why" part of that.

Steve

I'd love to know what they did - the manual for the 2920s just says:
"The two trunk group options (LACP and trunk) use SA/DA pairs for distributing outbound traffic
over trunked links"

and they don't offer any options (in the manual) for using a different hashing method for destination link selection.

Sounds like there is a hidden command in there...if you can find out what the command is, you will find out what the hashing method is, and then you will be able to work out how your traffic is flowing and then maybe figure out what changes you can try making to get your traffic to spread out a bit more over the physical links.

Just remember: Link Aggregation does *not* offer load balancing, it's just a distribution of traffic than can, in the right circumstances, look a bit like load balancing.
The place where Link Aggregation looks least like load balancing is where you use Link Aggregation between a single server and its network switch.

In other words, if the SourceAddress/DestinationAddress pair across your packets doesn't show much variation, then you'll get the least benefit from Link Aggregation.

Hi

I have just looked at the config file for my 2920 switches, and i found this line in it

trunk-load-balance L4-based

this what they changed, from L3 to L4 but i can't remember the actual command. this allowed me to get the extra 2Gbps.

to be honest, i don't really know much about switch config, my initial idea was to have two switches either side unstacked and untrunked... each sync channel was going to use its own path, but HP sales convinced me this wasn't going to work and sold me the stacking modules and cable, which then allowed me to make a trunk to get me 20Gbps but it hasn't really turned out to be the case.... it is enough bandwidth for now but i am very confused on how i would get more bandwidth when the time comes. i have two fibres linking each building, each with 4 cores in it, only 2 cores of each cable are currently wired up. so we have space capacity in the fibres but I'd be hesitant to buy more modules and wire them up if we can't exceed this 12Gbps limit.

"L4-based" would mean they are using the TCP port number in their hashing algorithm.

You could have a look at your apps and see if you can introduce additional port numbers somehow, you might be able to spread the traffic a bit more.

Don't forget that your limit on throughput could be due to disk read/write speed or something, not necessarily the physical links.

thanks, i don't know how to change stuff like that, networking isnt my strong point i'm afraid. i need to go on a course to learn more about it, my main focus is Windows based stuff. 

I am hoping that disk throughput doesnt come into the equation as i am using the NTTtcp tool to test directly from one NIC to another through the switches.... I assume that doesnt touch the disk?

I have two 10 Gbps NICS, when i test 1 against the other (in the other building) i get 10Gbps, when i try to test both at the same time i got around 5Gbps on each (still 10Gbps total) then when HP made the change i managed to get a total of 12Gbps - that's as far as my testing goes and my knowledge at this stage. If I wanted to make major changes to the switches i think i may need to get an expert in, but its not a concern right now until we start saturating that bandwidth a bit more.

thanks for the advice

Steve