We have two Windows Server 2012 machines with Intel NetEffect NE020 RDMA NICs fitted. Previously we had SMB Direct working between these two machine - we just transferred files and SMB Direct was used automatically. However since then one of the machines has been upgraded to Windows Server 2012 r2 and we have made some changes to the network, and now SMB Direct is no longer working.

I've followed all of the setup steps in Jose Barreto's blog: http://blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-smb-direct-smb-over-rdma-and-the-intel-neteffect-ne020-card-using-iwarp-step-by-step.aspx.  I believe I have all of the correct firewall rules, but I've disabled the firewall on both machines to eliminate it. I've also cut out the switch and directly connected the two NICs, with a couple of different cables. None of these changes makes a difference. I've enabled various logs, all of the SMBDirect and SMBClient logs, including ObjectStateDiagnostic as suggested in Jose Barreto's blog, and none of them yields and obviously relevant messages.

Message Analyzer gives a hint at the problem - I can see the SMB Direct negotiate request leave the client, it's received by the server and the server sends a response with success status. However the client never sees the responses, and appears to time out after ~30s, issuing an NDK disconnect. In the interim the file transfer completes over ordinary SMB3.0. I've repeated this many times with the same result. Most of the time the r2 machine has been acting as the client, but it fails in the same way if I swap client and server.

One other quirk I noticed in Message Analyzer was that the SMB Direct sockets on the server have the same local and remote IP addresses, the IP address of the server. On the client the sockets look correct. I've no idea whether this is expected, a bug in Message Analyzer or maybe the root of the problem.

Any ideas on how this problem could be diagnosed further, or what the problem might be?

• Edited by Andrew Royal Monday, February 16, 2015 9:35 AM Wrong link

I've upgraded the server to 2012 r2, reinstalled the client which was already running r2, and the problem persists. However Message Analyzer shows that behaviour has changed a little - now the requests are getting lost, the server times out after ~5s, cancels the accept and issues an NDK disconnect. Given that we previously had two 2012 machines working, my interpretation of this is that 2012 r2 is rejecting incoming SMB Direct packets in my setup for some reason. But why that should be, or what's particular to my setup which could be causing this, remains a mystery.

I've already used Message Analyzer which, as I understand it, supersedes Network Monitor. As I said in the original post, Message Analyzer shows the SMB Direct negotiate response leave the server but it never arrives at the client. When I upgraded the server to 2012 r2, the SMB Direct negotiate request leaves the client, but never arrives at the server.

In both cases removing the switch and directly connecting the two machines doesn't solve the problem. However with a switch I was able to use port mirroring to a third machine which does not have an RDMA NIC and packet capture with tshark. That shows that the missing packets are being sent to the port connected to the destination NIC, so the problem would appear to be on the receive side with packets getting lost somewhere in the lower layers before reaching the SMB Direct layer. I've tried using trace scenarios which show the lower layers, but nothing of interest shows up. Having said that I'm not sure how much of the lower layers is visible to Message Analyzer, from what I can tell much of RDMA is hidden from upper layers by the NIC.

This week I downgraded both machines to Windows Server 2012 Standard and SMB Direct now works. I think this proves that the issue is specific to 2012 r2.

Could you recommend any specific ETW providers to look at in Message Analyzer?

• Edited by Andrew Royal 23 hours 5 minutes ago