Exchange 2013 Online mode slow performance

Hello,

We recently moved mailboxes (180 or so) from Exchange 2007 to Exchange 2013.  We are noticing that Outlook navigation seems to be very slow compared to the last server.  Changing folder views, switching from Inbox to Calendar, Deleting items, etc.  Also we used to be able to send 40MB HTML emails very quickly.  This now takes about 5 min just to open a reply. (These are tripwire reports.)  Most users are using Outlook in Online mode on the new server and were using Online mode on the old server

Environment: Exchange 2013 SP1 on Server 2012r2 Standard (6 cores 24 gigs of RAM, VMWare VM.)  Old server is Exchange 2007 8.3.348.2 on Server 2008x64 Standard SP2 (8 cores 12 gigs of RAM. Physical.)

I have a ticket open with MS but didn't really get anywhere after 9 hours yesterday.

Thanks.

June 19th, 2014 5:31pm

Please post if have any information.
Free Windows Admin Tool Kit Click here and download it now
June 20th, 2014 1:49pm

The first thing that popped into my head was you changed from physical to a VM. Are you sure your VM Cluster is not over provisioned? I see you have 6 Cores assigned to the VM do you know about CPU ready time? Also RAM over provisioning can cause issues as similar to CPU ready. Perhaps its also disk/SAN(if you have one) latency as well.

RAM Bolloning:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003470

CPU ready: http://www.sqlskills.com/blogs/jonathan/cpu-ready-time-in-vmware-and-how-to-interpret-its-real-meaning/

Hope this helps!

 


June 20th, 2014 3:06pm

 My ticket is still open with Microsoft.  After about 15.5 hours with Microsoft we still don't know why Outlook using Online mode is faster to connect to Exchange 2007 than it is to connect to Exchange 2013.
Free Windows Admin Tool Kit Click here and download it now
June 20th, 2014 5:45pm

The first thing that popped into my head was you changed from physical to a VM. Are you sure your VM Cluster is not over provisioned? I see you have 6 Cores assigned to the VM do you know about CPU ready time? Also RAM over provisioning can cause issues as similar to CPU ready. Perhaps its also disk/SAN(if you have one) latency as well.

RAM Bolloning:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003470

CPU ready: http://www.sqlskills.com/blogs/jonathan/cpu-ready-time-in-vmware-and-how-to-interpret-its-real-meaning/

Hope this helps!

 


June 20th, 2014 6:06pm

RAM is thin provisioned but 16 gigs of it comes up as private.  57% of 131 gigs of RAM is used on the host.  17% of the CPU is used on the host.  So I don't think it's an over provision issue.
Free Windows Admin Tool Kit Click here and download it now
June 21st, 2014 12:24am

1. Ensure you have sized the guest correctly:

http://blogs.technet.com/b/exchange/archive/2013/05/14/released-exchange-2013-server-role-requirements-calculator.aspx

2. Verify that the BIOS on the vmhost is set to high performance

http://blogs.vmware.com/vsphere/2014/01/bios-power-policies-affect-performance.html

June 21st, 2014 4:40am

As Andy said- size your Exchange using Role Req. Calc.

On top of that i will make sure that you have not enabled the hyperthreading , if so then you need to disabled.

and also

guest virtual machine share a fixed number of logical processors in the physical system. Exchange supports a virtual processor-to-logical processor ratio no greater than 2:1, although we recommend a ratio of 1:1.

http://technet.microsoft.com/en-us/library/jj619301%28v=exchg.150%29.aspx#BKMK_Prereq

Free Windows Admin Tool Kit Click here and download it now
June 21st, 2014 6:37am

As Andy said- size your Exchange using Role Req. Calc.

On top of that i will make sure that you have not enabled the hyperthreading , if so then you need to disabled.

and also

guest virtual machine share a fixed number of logical processors in the physical system. Exchange supports a virtual processor-to-logical processor ratio no greater than 2:1, although we recommend a ratio of 1:1.

http://technet.microsoft.com/en-us/library/jj619301%28v=exchg.150%29.aspx#BKMK_Prereq

June 21st, 2014 6:06pm

Still no luck.  And I've been on the phone with MS each day.

The VM has been scaled down to 4 CPU's and 18 gigs of RAM.  It is currently only using 13 of the 18 gigs.  I also changed the CPU's to 4 sockets instead of 2 sockets with 2 cores.  We also set the memory reservation to 18 gigs.

The vCPU allocation is much higher than 2:1, but VMWare handles CPU loads different than Hyper-V.  Currently it is at less than half capacity.  It's using 7,602 MHz out of 8X2.599 GHz

Free Windows Admin Tool Kit Click here and download it now
June 25th, 2014 10:17pm

how many mailboxes and what is the avg size of the mailbox?

do you see in the task manager which application is consuming the resources?

June 25th, 2014 11:49pm

Still no luck.  And I've been on the phone with MS each day.

The VM has been scaled down to 4 CPU's and 18 gigs of RAM.  It is currently only using 13 of the 18 gigs.  I also changed the CPU's to 4 sockets instead of 2 sockets with 2 cores.  We also set the memory reservation to 18 gigs.

The vCPU allocation is much higher than 2:1, but VMWare handles CPU loads different than Hyper-V.  Currently it is at less than half capacity.  It's using 7,602 MHz out of 8X2.599 GHz


Ok, regardless, the vCPU ratio should not be higher than 2:1. Have you asked VMware about this?
Free Windows Admin Tool Kit Click here and download it now
June 26th, 2014 3:17am

Microsoft.Exchange.Store.Worker is consuming the most RAM at 1.2 and 1.4 gigs (there are 2 of these processes.)

Mailbox count is 210 and the average size is probably 350 MB.

What is the reason for the vCPU ratio if systems like VMware do not segregate CPU workload?

June 26th, 2014 6:49pm

Its about what is supported and recommended:

http://technet.microsoft.com/en-us/library/jj619301(v=exchg.150).aspx

http://www.vmware.com/files/pdf/Exchange_2013_on_VMware_Best_Practices_Guide.pdf

I would also ensure the host power options are set to "High Performance"

Free Windows Admin Tool Kit Click here and download it now
June 26th, 2014 7:49pm

I think disk usage is causing performance issues. Did you check Disk Queue length and Average Disk space usage on Exchange 2007 and Exchange 2013?

What is the storage that you use on VMware? Try with sharing file on Exchange 2007 and Exchange 2013, and see how much times it takes to copy to client machine.

I hope that it leads investigation to right direction.

June 26th, 2014 7:50pm

We are using a Compellent SAN.  We have 1 datastore that houses the C and D drive of our Exchange 2013 server.  These are the only 2 volumes on that data store and it is in tier 1 storage.

Disk queue length 2013: C=2.3 and lower / D=.1 and lower

Disk queue length 2007: C and D are barely above 0

Would setting a CPU reservation equate to correcting the ratio.  Might be easier than flushing the host of most VM's

Free Windows Admin Tool Kit Click here and download it now
June 26th, 2014 9:08pm

We are using a Compellent SAN.  We have 1 datastore that houses the C and D drive of our Exchange 2013 server.  These are the only 2 volumes on that data store and it is in tier 1 storage.

Disk queue length 2013: C=2.3 and lower / D=.1 and lower

Disk queue length 2007: C and D are barely above 0

Would setting a CPU reservation equate to correcting the ratio.  Might be easier than flushing the host of most VM's

I Don't think reservations are recommended. I would read that white paper or contact VMware support for the recommended settings.

June 27th, 2014 2:31am

You may need to run ESXTOP to see where the performance bottleneck is.

It is clear that issue is most likely with environment outside Exchange. It could be either SAN or VMware.

Free Windows Admin Tool Kit Click here and download it now
June 27th, 2014 5:34am

Thanks for all the replies.

I have a few things I can try at this point.  I just need to get the ok to do so and schedule them.

1. Clear the VMhost of other VM's to reduce the vCPU ratio

2. Try setting the power management to High performance

3. Disable offload features on the VMXNet3 NIC. My co-worker found this one.  I also noticed when moving the mailboxes it took about 9 hours for 60 gigs.  And both servers have gig connections between them.  Last night when trying to move everyone back the old server the send/receive stats on the new server (2012r2) were about 3.2 MBs

http://virtualrealization.blogspot.com/2013/11/vmxnet3-and-windows-operating-system.html

 

 

June 27th, 2014 4:39pm

60GB in 9 hrs., this is about 15Mbps (or 1.8 MBps). I would still recommend you to check Disk I/O performance especially on VMware. Are the disks on VMware thick or thin?
Free Windows Admin Tool Kit Click here and download it now
June 27th, 2014 7:19pm

The disks in VMware are thin provisioned. 

2013 Server: The SAN volume only the C and D drive for the 2013 server on it.  Both are separate VMDK files

2007 Server: We use a passthough disk from the SAN.  Meaning we map a drive right to the SAN from the BIOS on the 2007 server. No VHD or VMDK files here.

June 30th, 2014 11:53pm

We are experiencing the similar issue with Exchange 2013 SP1 on VMware ESXi4 and Outlook 2010.

OWA runs just fine, but Outlook in online mode is unacceptably slow.

There is no processor, memory or storage bottleneck. Servers were sized for a greater load and the configuration was checked with Jetstress tool.

Free Windows Admin Tool Kit Click here and download it now
July 11th, 2014 10:27am

The disks in VMware are thin provisioned. 

2013 Server: The SAN volume only the C and D drive for the 2013 server on it.  Both are separate VMDK files

2007 Server: We use a passthough disk from the SAN.  Meaning we map a drive right to the SAN from the BIOS on the 2007 server. No VHD or VMDK files here.

BriGi86

Could you check the performance when you are running Outlook in safe mode?

outlook.exe /safe

July 11th, 2014 11:12am

We ended up moving all users back to 2007.  We may wait for another service pack to test this again.
Free Windows Admin Tool Kit Click here and download it now
July 18th, 2014 9:05pm

We have many similarities in our environment and yours (VMWare/Compellent/Echange 2010 Online mode/etc.) and are seeing the exact same issues with performance.  The biggest difference that I see is that our Exchange 2010 environment was virtual and not physical servers.  Outlook 2010 and Exchange 2010 in online mode extremely responsive.  Now that we are running Exchange 2013 the experience is much worse.  OWA is very responsive only Outlook has the issue.  We can't run in cached mode because we have a VDI environment.  We've had a case opened with Microsoft for two months plus and are trying to have it escalated to a US based Exchange team.  Would love to share notes with you and find out if you'd be ok with sharing your case number that you had opened. 

August 19th, 2014 5:05pm

Hi everyone,

Here we have exactly the same thing. We did the same migration from Exchange 2k7 to Exchange 2013. We run on a VMware 5.5 infrastructure. We have a lot of users running on a VM, but I can also tell you that I'm experiencing the problem even on a physical PC. So the problem about the client side infrastructure is maybe not that much related.

 I also found an old thread talking about a known issue with the grouping option in Outlook and online mode, but this is supposed to be only for Exchange and Outlook 2007, so i'm not able to prove this problem is exactly and only related to this issue. however, I tried to disable the grouping option for some folder in my Outlook and I saw an improvement immediately. Here is the thread talking about this issue and the knowledge base associated from Microsoft:

http: //blogs.technet.com/b/mikelag/archive/2008/08/31/slow-outlook-online-mode-performance.aspx

http: //support.microsoft.com/kb/948828

Sorry, I cannot post the real link since my account is not verified yet... simply remove the space.

If anyone finally found an answer please post-it, thanks.

Free Windows Admin Tool Kit Click here and download it now
August 22nd, 2014 8:21pm

We are having the same issue on two properly sized physical Exchange 2013 servers.  We opened a case with Microsoft over six months ago and are still waiting for resolution.

August 26th, 2014 6:35pm

Hi everyone,

After some search on our environment we found that it seem to come from the connexion type (still not sure at all but...). Exchange 2007 was supporting Tcp/IP connexions with Outlook, but now with Exchange 2013, this isn't possible if i'm not wrong. I did a short test since I still have access to my 2k7 Exchange server. I forced my Outlook connected to our old Exchange server (2k7) to use the HTTP connexion type first to try to reproduce similar condition that we have on our 2013. As we expected we had pretty similar symptoms of slowness in Outlook. Then I turned off the setting to use http first and retry with the same Outlook and I retreived the fast and sweet performance we were having before our migration. I'm not an Exchange expert but this sound for me a good explication (theory). Our next step will be to update our Exchange server with Cumulative update 5 for our Exchange 2013 server. We hope this will help us. If anyone can confirm this theory, it may help Microsoft or other people to find a solution.

Free Windows Admin Tool Kit Click here and download it now
August 27th, 2014 10:29pm

Hi there,

Same here, I've got a multisite architecture and all the users who have been moved from 2007 to 2013 are experiencing the same slowness in online mode. 

In owa the problem does not exists.

By proxying the access trough 2013 cas to mailbox on 2007 the issue does not exists.

This behavior is happening just in coexistence, I did some new installation and the online mode is fast.

Any idea?

August 28th, 2014 10:59am

Any news?

Does anybody tried with the CU6?

Free Windows Admin Tool Kit Click here and download it now
August 29th, 2014 10:52am

I too would be interested if CU6 resolves this problem. I experience this periodically and it is hard to isolate.
September 9th, 2014 10:30pm

CU6 does not resolve this.
Take a look at this post:

http://social.technet.microsoft.com/Forums/office/en-US/0d45b1b0-3047-4666-ad04-217e98ed8823/slow-online-mode-browsing?forum=exchangesvrclients

  • Proposed as answer by MasterKic Tuesday, September 16, 2014 5:06 PM
  • Unproposed as answer by MasterKic Tuesday, September 16, 2014 5:07 PM
Free Windows Admin Tool Kit Click here and download it now
September 10th, 2014 9:09am

CU6 does not resolve this.
Take a look at this post:

http://social.technet.microsoft.com/Forums/office/en-US/0d45b1b0-3047-4666-ad04-217e98ed8823/slow-online-mode-browsing?forum=exchangesvrclients

  • Proposed as answer by MasterKic Tuesday, September 16, 2014 5:06 PM
  • Unproposed as answer by MasterKic Tuesday, September 16, 2014 5:07 PM
September 10th, 2014 12:09pm

OK We found something very very interesting about this. We found a solution that worked for us. Have a look at this other thread:

http:// community.spiceworks.com/topic/571571-outlook-slow-after-migrating-to-exchange-2013

Since these changes can affect the entire Workstation, I suggest you try it and read the entire post before doing it.

TcpAckFrequency:

http:// support.microsoft.com/kb/328890/fr

I'm sorry the link are not valid since I'm not able to post it correctly right now but, I hope this will help many other people. Thx.

  • Marked as answer by BriGy86 Wednesday, February 18, 2015 5:07 PM
Free Windows Admin Tool Kit Click here and download it now
September 16th, 2014 5:13pm

OK We found something very very interesting about this. We found a solution that worked for us. Have a look at this other thread:

http:// community.spiceworks.com/topic/571571-outlook-slow-after-migrating-to-exchange-2013

Since these changes can affect the entire Workstation, I suggest you try it and read the entire post before doing it.

TcpAckFrequency:

http:// support.microsoft.com/kb/328890/fr

I'm sorry the link are not valid since I'm not able to post it correctly right now but, I hope this will help many other people. Thx.

  • Marked as answer by BriGy86 Wednesday, February 18, 2015 5:07 PM
September 16th, 2014 5:13pm

OK We found something very very interesting about this. We found a solution that worked for us. Have a look at this other thread:

http:// community.spiceworks.com/topic/571571-outlook-slow-after-migrating-to-exchange-2013

Since these changes can affect the entire Workstation, I suggest you try it and read the entire post before doing it.

TcpAckFrequency:

http:// support.microsoft.com/kb/328890/fr

I'm sorry the link are not valid since I'm not able to post it correctly right now but, I hope this will help many other people. Thx.

Free Windows Admin Tool Kit Click here and download it now
September 16th, 2014 8:13pm

We are having the EXACT same issue. Exchange 2013 single server environment running on Server 2012 R2, updated to CU6. Running on VMWare 5.5 (latest updates) with VMXNET3 drivers. Compellent SAN. Also, all users migrated from Exchange 2007. Clients are running in ONLINE MODE due to VDI. OWA works flawlessly. All clients experiencing VERY HIGH RPC latency (4000+ ms when running RPCPING).

Ticket opened with Microsoft over 6 months ago and has been escalated MANY times. Network team points fingers at Exchange, Exchange team points fingers at Outlook team, and Outlook team points fingers back stating it's not their issue.

FYI, Outlook runs GREAT when installed ON the Exchange server (less than 15ms RPC latency). I've even tried installing Outlook on another VM on the same VMWare host and in the same subnet so that all networking stays on the host.

Subscribing to this thread...

November 18th, 2014 1:42am

This is from the SpiceWorks link posted above. I have not tried it though.

Looks like there is a registry key that can be done on each client or a fix for VMware. https://communities.vmware.com/message/2439033I

I am going to try rolling the vmwaretools back to 5.1 RU1 and move a problem user to the new server.  If this works I will be marking MasterKic's reply as the answer as well as this one.

  • Marked as answer by BriGy86 Wednesday, February 18, 2015 5:06 PM
Free Windows Admin Tool Kit Click here and download it now
December 16th, 2014 3:00pm

This is from the SpiceWorks link posted above. I have not tried it though.

Looks like there is a registry key that can be done on each client or a fix for VMware. https://communities.vmware.com/message/2439033I

I am going to try rolling the vmwaretools back to 5.1 RU1 and move a problem user to the new server.  If this works I will be marking MasterKic's reply as the answer as well as this one.

  • Marked as answer by BriGy86 Wednesday, February 18, 2015 5:06 PM
December 16th, 2014 3:00pm

This is from the SpiceWorks link posted above. I have not tried it though.

Looks like there is a registry key that can be done on each client or a fix for VMware. https://communities.vmware.com/message/2439033I

I am going to try rolling the vmwaretools back to 5.1 RU1 and move a problem user to the new server.  If this works I will be marking MasterKic's reply as the answer as well as this one.

Free Windows Admin Tool Kit Click here and download it now
December 16th, 2014 6:00pm

BriGy86: The link to vmware communities you posted is not working. Can you please provide another link?

EDIT: Found the article in question! https://communities.vmware.com/message/2439033?tstart=0

Thanks!


  • Edited by JBHorne Friday, December 19, 2014 8:11 PM Update link
December 19th, 2014 8:04pm

BriGy86: The link to vmware communities you posted is not working. Can you please provide another link?

EDIT: Found the article in question! https://communities.vmware.com/message/2439033?tstart=0

Thanks!


  • Edited by JBHorne Friday, December 19, 2014 8:11 PM Update link
Free Windows Admin Tool Kit Click here and download it now
December 19th, 2014 11:04pm

Sorry for the late response.  I have since gotten the tools rolled back

TAKE A SNAPSHOT!  removing the tools resulted in the old exchange server inoperable for about a day.  Then all of a sudden it started responding (This is after several reboots.)

After getting them rolled back I ran into some other mail flow issues while testing so I have not completely resolved this issue yet.

January 30th, 2015 6:07pm

After moving some users to the new machine we are not seeing any issues. (The one with the tools version rolled back.)

While building the new server I made sure to install the rolled back version of VMware tools and did not get any errors during the install or SP1 install.  The previous server I built had horrendous issues with the install.

Free Windows Admin Tool Kit Click here and download it now
March 9th, 2015 3:17pm

I'm having similar problems with online mode using HTTPS (preparing for our Exchange 2013 migration).   Our Outlook Anywhere and RPC Client Access array host names are the same, so, the suggestion from Brian Day at Microsoft is to put all your clients into HTTPS mode before cutting over DNS to Ex2013.

If we attempt to have our clients (Outlook 2010 and 2013) connect using HTTPS (Outlook Anywhere) versus MAPI in Exchange 2010, the experience is poor at times.   Exchange 2010 is all physical.    We require online mode because of a VDI environment.    

I've been working with Microsoft on this issue for two to three months without any progress.  We've ruled a lot out, but, haven't found the issue.      We've tried the TCPAck setting without any change.

If anyone finds the smoking gun, please post on this.

April 22nd, 2015 5:41pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics