CAS restart and Mailbox database dismount unexpectedly somtimes

Hi all,

In my environment I have 2 Exchange server 2013 (CAS + Mailbox Role both) CU2 : EX01 & EX02 in DAG01. Sometimes CAS service is restarted or Mailbox Database copy on Exchange servers is dismounted unexpectedly. For example today:
I have "Mailbox Database 01" copies on EX01 & EX02

Get-MailboxDatabaseCopyStatus "Mailbox Database 01"
Name                                Status
Mailbox Database 01\EX01            Healthy
Mailbox Database 01\EX02            Mounted

"Mailbox Database 01" copy on EX02 is dismounted, of course the other copy on EX01 is mounted automatically

Log Name:      Application
Source:        MSExchangeRepl
Date:          4/3/2015 2:32:19 PM
Event ID:      2153
Task Category: Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX02.mydomain.com
Description:
The log copier was unable to communicate with server 'EX01.mydomain.com'. The copy of database 'Mailbox Database 01\EX02' is in a disconnected state. The communication error was: An error occurred while communicating with server 'EX01'. Error: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. The copier will automatically retry after a short delay.

CAS service on EX01 , The Microsoft Exchange Mailbox Assistants service  on EX02 are restarted

Log Name:      System
Source:        Service Control Manager
Date:          4/3/2015 3:25:49 PM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange RPC Client Access service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/3/2015 3:25:51 PM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange RPC Client Access service entered the running state.

Log Name:      System
Source:        Service Control Manager
Date:          4/3/2015 3:25:54 PM
Event ID:      7032
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Service Control Manager tried to take a corrective action (Restart the service) after the unexpected termination of the Microsoft Exchange RPC Client Access service, but this action failed with the following error: 
An instance of the service is already running.

Log Name:      System
Source:        Service Control Manager
Date:          4/3/2015 3:30:28 PM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX02.mydomain.com
Description:
The Microsoft Exchange Mailbox Assistants service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/3/2015 3:30:30 PM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      EX02.mydomain.com
Description:
The Microsoft Exchange Mailbox Assistants service entered the running state.

Log Name:      System
Source:        Service Control Manager
Date:          4/3/2015 3:30:33 PM
Event ID:      7032
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX02.mydomain.com
Description:
The Service Control Manager tried to take a corrective action (Restart the service) after the unexpected termination of the Microsoft Exchange Mailbox Assistants service, but this action failed with the following error: 
An instance of the service is already running.

On EX01 I can see many 1309 , 2009 , 6002 , 4999 in Event Viewer on both servers, and they just used half of RAM

I'm going to upgrade to SP1 this weekend but please help me to figure out really reason of this issue, limit of max connections on Exchange/Window server , or Disk IO or something ?

Thanks for your help.

April 3rd, 2015 6:16am

Hi Jack,

Do you happen to have any DAG FileShare Witness server.

What does below command say:

Get-ClusterQuorum | fl Get-ClusterNode | ft Name,ID,DynamicWeight,NodeWeight,State -auto

Get-Cluster | fl *Wit*

Looks like Managed availablity is kicking in, you need to dig in and check some probes maybe to figure this out.

EventLogs:

Event Viewer > Application and Services Logs > Microsoft > Exchange >

Use below cmdlets to check health:

Cmdlet                

Get-ServerHealth

Get-HealthReport

Free Windows Admin Tool Kit Click here and download it now
April 3rd, 2015 7:54am

If you plan on updating, I would recommend going past SP1 and applying either CU7 or CU8. These are the currently supported patch levels for Exchange 2013.  This is helpful for 2 reasons, first SP1(really called CU4 but the marketing folks thought sp1 sounded better) isn't supported any longer (Microsoft only supports something like 2 CU's back). Secondly, if updating to the latest binaries don't resolve the issue, then you might need to open a case with Microsoft to have them investigate, and they will most certainly want you to upgrade to a later CU before even looking at it the issue.
April 3rd, 2015 8:54am

I have DAG Witness server

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\oldwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

[PS] C:\Windows\system32>Get-ClusterNode | ft Name,ID,DynamicWeight,NodeWeight,State -auto

Name Id      DynamicWeight NodeWeight State
ex01 0001                             Up
ex02 0002                             Up

[PS] C:\Windows\system32>Get-Cluster | fl *Wit*

WitnessDatabaseWriteTimeout : 300
WitnessRestartInterval      : 15

I should explain this, I changed Witness server in DAG01 configuration today and I did successfully, C:\Witness folder is created and shared as "DAG01.mydomain.com" on newwitness server  automatically. I don't know why Get-ClusterQuorum still show QuorumResource as oldwitness instead newwitness.

==> I changed QuorumResource by using Failover Cluster Manager in Exchange server and it shows right QuorumResource now

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\newwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

The result of health check on EX01 and EX02


[PS] C:\Windows\system32>Get-ServerComponentState ex01 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active

[PS] C:\Windows\system32>Get-ServerComponentState ex02 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active


Free Windows Admin Tool Kit Click here and download it now
April 3rd, 2015 12:20pm

Thanks Hinte,
I read from this topic : https://social.technet.microsoft.com/Forums/office/en-US/8cf94d7a-d36b-4b5d-a32e-a9aa538c427e/exchange-2013-upgrade-cu2-to-cu7?forum=exchangesvrdeploy
They said that it's better we have exchange 2013 SP1 before we install the latest CU ? I don't know if it's safe to go from CU2 to CU7 directly ?
April 3rd, 2015 12:31pm

Thanks Hinte,
I read from this topic : https://social.technet.microsoft.com/Forums/office/en-US/8cf94d7a-d36b-4b5d-a32e-a9aa538c427e/exchange-2013-upgrade-cu2-to-cu7?forum=exchangesvrdeploy
They said that it's better we have exchange 2013 SP1 before we install the latest CU ? I don't know if it's safe to go from CU2 to CU7 directly ?
All the updates are cumulative, meaning you can go from any version to the latest one.  I really don't think ot matters which path you actually take.  If it was me and i was scheduling after hours downtime, I would just go straight to CU7 (I like my off work hours to stay off work as much as possible). If you're worried that may be too big of a jump then by all means do the double hop (although if you do the double hop, i wouldn't do both the same night just incase there is a failure at some point you can pinpoint what exactly caused the failure).  I don't recall seeing any issues jumping from a presp1 CU to a more recent CU.
Free Windows Admin Tool Kit Click here and download it now
April 3rd, 2015 1:19pm

I have DAG Witness server

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\oldwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

[PS] C:\Windows\system32>Get-ClusterNode | ft Name,ID,DynamicWeight,NodeWeight,State -auto

Name Id      DynamicWeight NodeWeight State
ex01 0001                             Up
ex02 0002                             Up

[PS] C:\Windows\system32>Get-Cluster | fl *Wit*

WitnessDatabaseWriteTimeout : 300
WitnessRestartInterval      : 15

I should explain this, I changed Witness server in DAG01 configuration today and I did successfully, C:\Witness folder is created and shared as "DAG01.mydomain.com" on newwitness server  automatically. I don't know why Get-ClusterQuorum still show QuorumResource as oldwitness instead newwitness.

==> I changed QuorumResource by using Failover Cluster Manager in Exchange server and it shows right QuorumResource now

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\newwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

The result of health check on EX01 and EX02


[PS] C:\Windows\system32>Get-ServerComponentState ex01 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active

[PS] C:\Windows\system32>Get-ServerComponentState ex02 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active


  • Edited by Jack Chuong Saturday, April 04, 2015 2:14 AM
April 3rd, 2015 4:17pm

I have DAG Witness server

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\oldwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

[PS] C:\Windows\system32>Get-ClusterNode | ft Name,ID,DynamicWeight,NodeWeight,State -auto

Name Id      DynamicWeight NodeWeight State
ex01 0001                             Up
ex02 0002                             Up

[PS] C:\Windows\system32>Get-Cluster | fl *Wit*

WitnessDatabaseWriteTimeout : 300
WitnessRestartInterval      : 15

I should explain this, I changed Witness server in DAG01 configuration today and I did successfully, C:\Witness folder is created and shared as "DAG01.mydomain.com" on newwitness server  automatically. I don't know why Get-ClusterQuorum still show QuorumResource as oldwitness instead newwitness.

==> I changed QuorumResource by using Failover Cluster Manager in Exchange server and it shows right QuorumResource now

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\newwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

The result of health check on EX01 and EX02


[PS] C:\Windows\system32>Get-ServerComponentState ex01 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active

[PS] C:\Windows\system32>Get-ServerComponentState ex02 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active


  • Edited by Jack Chuong Saturday, April 04, 2015 2:14 AM
Free Windows Admin Tool Kit Click here and download it now
April 3rd, 2015 4:17pm

I have DAG Witness server

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\oldwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

[PS] C:\Windows\system32>Get-ClusterNode | ft Name,ID,DynamicWeight,NodeWeight,State -auto

Name Id      DynamicWeight NodeWeight State
ex01 0001                             Up
ex02 0002                             Up

[PS] C:\Windows\system32>Get-Cluster | fl *Wit*

WitnessDatabaseWriteTimeout : 300
WitnessRestartInterval      : 15

I should explain this, I changed Witness server in DAG01 configuration today and I did successfully, C:\Witness folder is created and shared as "DAG01.mydomain.com" on newwitness server  automatically. I don't know why Get-ClusterQuorum still show QuorumResource as oldwitness instead newwitness.

==> I changed QuorumResource by using Failover Cluster Manager in Exchange server and it shows right QuorumResource now

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\newwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

The result of health check on EX01 and EX02


[PS] C:\Windows\system32>Get-ServerComponentState ex01 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active

[PS] C:\Windows\system32>Get-ServerComponentState ex02 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active


  • Edited by Jack Chuong Saturday, April 04, 2015 2:14 AM
April 3rd, 2015 4:17pm

I have DAG Witness server

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\oldwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

[PS] C:\Windows\system32>Get-ClusterNode | ft Name,ID,DynamicWeight,NodeWeight,State -auto

Name Id      DynamicWeight NodeWeight State
ex01 0001                             Up
ex02 0002                             Up

[PS] C:\Windows\system32>Get-Cluster | fl *Wit*

WitnessDatabaseWriteTimeout : 300
WitnessRestartInterval      : 15

I should explain this, I changed Witness server in DAG01 configuration today and I did successfully, C:\Witness folder is created and shared as "DAG01.mydomain.com" on newwitness server  automatically. I don't know why Get-ClusterQuorum still show QuorumResource as oldwitness instead newwitness.

==> I changed QuorumResource by using Failover Cluster Manager in Exchange server and it shows right QuorumResource now

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\newwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

The result of health check on EX01 and EX02


[PS] C:\Windows\system32>Get-ServerComponentState ex01 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active

[PS] C:\Windows\system32>Get-ServerComponentState ex02 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active


  • Edited by Jack Chuong Saturday, April 04, 2015 2:14 AM
Free Windows Admin Tool Kit Click here and download it now
April 3rd, 2015 4:17pm

I have DAG Witness server

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\oldwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

[PS] C:\Windows\system32>Get-ClusterNode | ft Name,ID,DynamicWeight,NodeWeight,State -auto

Name Id      DynamicWeight NodeWeight State
ex01 0001                             Up
ex02 0002                             Up

[PS] C:\Windows\system32>Get-Cluster | fl *Wit*

WitnessDatabaseWriteTimeout : 300
WitnessRestartInterval      : 15

I should explain this, I changed Witness server in DAG01 configuration today and I did successfully, C:\Witness folder is created and shared as "DAG01.mydomain.com" on newwitness server  automatically. I don't know why Get-ClusterQuorum still show QuorumResource as oldwitness instead newwitness.

==> I changed QuorumResource by using Failover Cluster Manager in Exchange server and it shows right QuorumResource now

Get-ClusterQuorum | fl

Cluster        : DAG01
QuorumResource : File Share Witness (\\newwitness.mydomain.com\DAG01.mydomain.com)
QuorumType     : NodeAndFileShareMajority

The result of health check on EX01 and EX02


[PS] C:\Windows\system32>Get-ServerComponentState ex01 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active

[PS] C:\Windows\system32>Get-ServerComponentState ex02 | ft Component,State -Autosize

Component                   State
---------                   -----
ServerWideOffline          Active
HubTransport               Active
FrontendTransport          Active
Monitoring                 Active
RecoveryActionsEnabled     Active
AutoDiscoverProxy          Active
ActiveSyncProxy            Active
EcpProxy                   Active
EwsProxy                   Active
ImapProxy                  Active
OabProxy                   Active
OwaProxy                   Active
PopProxy                   Active
PushNotificationsProxy     Active
RpsProxy                   Active
RwsProxy                   Active
RpcProxy                   Active
UMCallRouter               Active
XropProxy                  Active
HttpProxyAvailabilityGroup Active
ForwardSyncDeamon          Active
ProvisioningRps            Active
MapiProxy                  Active


  • Edited by Jack Chuong Saturday, April 04, 2015 2:14 AM
April 3rd, 2015 4:17pm

I will go to CU7 directly, as I read here : http://blogs.technet.com/b/rmilne/archive/2014/12/09/exchange-2013-cu7-released.aspx
I should download and install Security Update For Exchange Server 2013 SP1 (KB3011140)  & Security Update For Exchange Server 2013 CU6 (KB3011140) prior to upgrading to CU7 , right ?
Free Windows Admin Tool Kit Click here and download it now
April 3rd, 2015 10:43pm

I'm not 100% both are part of CU7, but the sp1 fix is, and the post cu6 fix should be.  I would go straight cu7
April 3rd, 2015 11:53pm

My EX01 is upgrading to CU7, I put it in maintenance mode of course, EX02 still working normally, everything is fine. Now, Exchange services on EX01 is stopped in upgrading process I check my "Mailbox Database 01" edb file on with command

eseutil /mh "path to Mailbox Database 01 edb file on EX01"

            State: Dirty Shutdown
     Log Required: 4826173-4826278 (0x49a43d-0x49a4a6)
    Log Committed: 0-4826279 (0x0-0x49a4a7)
   Log Recovering: 4826279 (0x49a4a7)

What should I do ?

Free Windows Admin Tool Kit Click here and download it now
April 4th, 2015 7:06pm

Hi all,

After upgraded EX01 to CU7, I try to upgrade EX02 too but error happens when I run commands

setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms setup.exe /mode:upgrade /IAcceptExchangeServerLicenseTerms

Welcome to Microsoft Exchange Server 2013 Cumulative Update 7 Unattended Setup
Copying Files...
File copy complete. Setup will now collect additional information needed for
installation.

The Exchange Server setup operation didn't complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

Here content of ExchangeSetupLogs\ExchangeSetup.log

[04/05/2015 01:23:55.0704] [0] Starting Microsoft Exchange Server 2013 Cumulative Update 7 Setup
[04/05/2015 01:23:55.0704] [0] **********************************************
[04/05/2015 01:23:55.0704] [0] Local time zone: (UTC+07:00) Bangkok, Hanoi, Jakarta.
[04/05/2015 01:23:55.0704] [0] Operating system version: Microsoft Windows NT 6.1.7601 Service Pack 1.
[04/05/2015 01:23:55.0719] [0] Setup version: 15.0.1044.25.
[04/05/2015 01:23:55.0719] [0] Logged on user: mydomain\administrator.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='mode', Value='Upgrade'.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='iacceptexchangeserverlicenseterms', Value=''.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='sourcedir', Value='D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] RuntimeAssembly was started with the following command: '/mode:upgrade /IAcceptExchangeServerLicenseTerms /sourcedir:D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\v8.0, wasn't found.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14, wasn't found.
[04/05/2015 01:23:56.0780] [0] Copying Files...
[04/05/2015 01:23:56.0796] [0] Starting copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] Finished copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] File copy complete. Setup will now collect additional information needed for installation.
[04/05/2015 01:23:59.0775] [0] Assembly dll file location is C:\Windows\Temp\ExchangeSetup\Microsoft.Exchange.Setup.Console.dll
[04/05/2015 01:23:59.0838] [0] [ERROR] An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
[04/05/2015 01:23:59.0838] [0] CurrentResult SetupLauncherHelper.loadassembly:444: 1
[04/05/2015 01:23:59.0838] [0] The Exchange Server setup operation didn't complete.  More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
[04/05/2015 01:23:59.0838] [0] CurrentResult main.run:235: 1
[04/05/2015 01:23:59.0838] [0] CurrentResult setupbase.maincore:396: 1
[04/05/2015 01:23:59.0838] [0] End of Setup

April 4th, 2015 9:50pm

Hi all,

After upgraded EX01 to CU7, I try to upgrade EX02 too but error happens when I run commands

setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms setup.exe /mode:upgrade /IAcceptExchangeServerLicenseTerms

Welcome to Microsoft Exchange Server 2013 Cumulative Update 7 Unattended Setup
Copying Files...
File copy complete. Setup will now collect additional information needed for
installation.

The Exchange Server setup operation didn't complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

Here content of ExchangeSetupLogs\ExchangeSetup.log

[04/05/2015 01:23:55.0704] [0] Starting Microsoft Exchange Server 2013 Cumulative Update 7 Setup
[04/05/2015 01:23:55.0704] [0] **********************************************
[04/05/2015 01:23:55.0704] [0] Local time zone: (UTC+07:00) Bangkok, Hanoi, Jakarta.
[04/05/2015 01:23:55.0704] [0] Operating system version: Microsoft Windows NT 6.1.7601 Service Pack 1.
[04/05/2015 01:23:55.0719] [0] Setup version: 15.0.1044.25.
[04/05/2015 01:23:55.0719] [0] Logged on user: mydomain\administrator.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='mode', Value='Upgrade'.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='iacceptexchangeserverlicenseterms', Value=''.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='sourcedir', Value='D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] RuntimeAssembly was started with the following command: '/mode:upgrade /IAcceptExchangeServerLicenseTerms /sourcedir:D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\v8.0, wasn't found.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14, wasn't found.
[04/05/2015 01:23:56.0780] [0] Copying Files...
[04/05/2015 01:23:56.0796] [0] Starting copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] Finished copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] File copy complete. Setup will now collect additional information needed for installation.
[04/05/2015 01:23:59.0775] [0] Assembly dll file location is C:\Windows\Temp\ExchangeSetup\Microsoft.Exchange.Setup.Console.dll
[04/05/2015 01:23:59.0838] [0] [ERROR] An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
[04/05/2015 01:23:59.0838] [0] CurrentResult SetupLauncherHelper.loadassembly:444: 1
[04/05/2015 01:23:59.0838] [0] The Exchange Server setup operation didn't complete.  More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
[04/05/2015 01:23:59.0838] [0] CurrentResult main.run:235: 1
[04/05/2015 01:23:59.0838] [0] CurrentResult setupbase.maincore:396: 1
[04/05/2015 01:23:59.0838] [0] End of Setup
Don't mind, I fixed it , because of lock exe or something... I just have to extract and run CU7 from local server.


Free Windows Admin Tool Kit Click here and download it now
April 5th, 2015 1:48am

Hi all,

After upgraded EX01 to CU7, I try to upgrade EX02 too but error happens when I run commands

setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms setup.exe /mode:upgrade /IAcceptExchangeServerLicenseTerms

Welcome to Microsoft Exchange Server 2013 Cumulative Update 7 Unattended Setup
Copying Files...
File copy complete. Setup will now collect additional information needed for
installation.

The Exchange Server setup operation didn't complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

Here content of ExchangeSetupLogs\ExchangeSetup.log

[04/05/2015 01:23:55.0704] [0] Starting Microsoft Exchange Server 2013 Cumulative Update 7 Setup
[04/05/2015 01:23:55.0704] [0] **********************************************
[04/05/2015 01:23:55.0704] [0] Local time zone: (UTC+07:00) Bangkok, Hanoi, Jakarta.
[04/05/2015 01:23:55.0704] [0] Operating system version: Microsoft Windows NT 6.1.7601 Service Pack 1.
[04/05/2015 01:23:55.0719] [0] Setup version: 15.0.1044.25.
[04/05/2015 01:23:55.0719] [0] Logged on user: mydomain\administrator.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='mode', Value='Upgrade'.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='iacceptexchangeserverlicenseterms', Value=''.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='sourcedir', Value='D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] RuntimeAssembly was started with the following command: '/mode:upgrade /IAcceptExchangeServerLicenseTerms /sourcedir:D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\v8.0, wasn't found.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14, wasn't found.
[04/05/2015 01:23:56.0780] [0] Copying Files...
[04/05/2015 01:23:56.0796] [0] Starting copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] Finished copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] File copy complete. Setup will now collect additional information needed for installation.
[04/05/2015 01:23:59.0775] [0] Assembly dll file location is C:\Windows\Temp\ExchangeSetup\Microsoft.Exchange.Setup.Console.dll
[04/05/2015 01:23:59.0838] [0] [ERROR] An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
[04/05/2015 01:23:59.0838] [0] CurrentResult SetupLauncherHelper.loadassembly:444: 1
[04/05/2015 01:23:59.0838] [0] The Exchange Server setup operation didn't complete.  More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
[04/05/2015 01:23:59.0838] [0] CurrentResult main.run:235: 1
[04/05/2015 01:23:59.0838] [0] CurrentResult setupbase.maincore:396: 1
[04/05/2015 01:23:59.0838] [0] End of Setup
Don't mind, I fixed it , because of lock exe or something... I just have to extract and run CU7 from local server.


April 5th, 2015 1:48am

Hi all,

After upgraded EX01 to CU7, I try to upgrade EX02 too but error happens when I run commands

setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms setup.exe /mode:upgrade /IAcceptExchangeServerLicenseTerms

Welcome to Microsoft Exchange Server 2013 Cumulative Update 7 Unattended Setup
Copying Files...
File copy complete. Setup will now collect additional information needed for
installation.

The Exchange Server setup operation didn't complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

Here content of ExchangeSetupLogs\ExchangeSetup.log

[04/05/2015 01:23:55.0704] [0] Starting Microsoft Exchange Server 2013 Cumulative Update 7 Setup
[04/05/2015 01:23:55.0704] [0] **********************************************
[04/05/2015 01:23:55.0704] [0] Local time zone: (UTC+07:00) Bangkok, Hanoi, Jakarta.
[04/05/2015 01:23:55.0704] [0] Operating system version: Microsoft Windows NT 6.1.7601 Service Pack 1.
[04/05/2015 01:23:55.0719] [0] Setup version: 15.0.1044.25.
[04/05/2015 01:23:55.0719] [0] Logged on user: mydomain\administrator.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='mode', Value='Upgrade'.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='iacceptexchangeserverlicenseterms', Value=''.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='sourcedir', Value='D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] RuntimeAssembly was started with the following command: '/mode:upgrade /IAcceptExchangeServerLicenseTerms /sourcedir:D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\v8.0, wasn't found.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14, wasn't found.
[04/05/2015 01:23:56.0780] [0] Copying Files...
[04/05/2015 01:23:56.0796] [0] Starting copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] Finished copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] File copy complete. Setup will now collect additional information needed for installation.
[04/05/2015 01:23:59.0775] [0] Assembly dll file location is C:\Windows\Temp\ExchangeSetup\Microsoft.Exchange.Setup.Console.dll
[04/05/2015 01:23:59.0838] [0] [ERROR] An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
[04/05/2015 01:23:59.0838] [0] CurrentResult SetupLauncherHelper.loadassembly:444: 1
[04/05/2015 01:23:59.0838] [0] The Exchange Server setup operation didn't complete.  More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
[04/05/2015 01:23:59.0838] [0] CurrentResult main.run:235: 1
[04/05/2015 01:23:59.0838] [0] CurrentResult setupbase.maincore:396: 1
[04/05/2015 01:23:59.0838] [0] End of Setup
Don't mind, I fixed it , because of lock exe or something... I just have to extract and run CU7 from local server.


Free Windows Admin Tool Kit Click here and download it now
April 5th, 2015 1:48am

Hi all,

After upgraded EX01 to CU7, I try to upgrade EX02 too but error happens when I run commands

setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms setup.exe /mode:upgrade /IAcceptExchangeServerLicenseTerms

Welcome to Microsoft Exchange Server 2013 Cumulative Update 7 Unattended Setup
Copying Files...
File copy complete. Setup will now collect additional information needed for
installation.

The Exchange Server setup operation didn't complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

Here content of ExchangeSetupLogs\ExchangeSetup.log

[04/05/2015 01:23:55.0704] [0] Starting Microsoft Exchange Server 2013 Cumulative Update 7 Setup
[04/05/2015 01:23:55.0704] [0] **********************************************
[04/05/2015 01:23:55.0704] [0] Local time zone: (UTC+07:00) Bangkok, Hanoi, Jakarta.
[04/05/2015 01:23:55.0704] [0] Operating system version: Microsoft Windows NT 6.1.7601 Service Pack 1.
[04/05/2015 01:23:55.0719] [0] Setup version: 15.0.1044.25.
[04/05/2015 01:23:55.0719] [0] Logged on user: mydomain\administrator.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='mode', Value='Upgrade'.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='iacceptexchangeserverlicenseterms', Value=''.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='sourcedir', Value='D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] RuntimeAssembly was started with the following command: '/mode:upgrade /IAcceptExchangeServerLicenseTerms /sourcedir:D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\v8.0, wasn't found.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14, wasn't found.
[04/05/2015 01:23:56.0780] [0] Copying Files...
[04/05/2015 01:23:56.0796] [0] Starting copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] Finished copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] File copy complete. Setup will now collect additional information needed for installation.
[04/05/2015 01:23:59.0775] [0] Assembly dll file location is C:\Windows\Temp\ExchangeSetup\Microsoft.Exchange.Setup.Console.dll
[04/05/2015 01:23:59.0838] [0] [ERROR] An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
[04/05/2015 01:23:59.0838] [0] CurrentResult SetupLauncherHelper.loadassembly:444: 1
[04/05/2015 01:23:59.0838] [0] The Exchange Server setup operation didn't complete.  More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
[04/05/2015 01:23:59.0838] [0] CurrentResult main.run:235: 1
[04/05/2015 01:23:59.0838] [0] CurrentResult setupbase.maincore:396: 1
[04/05/2015 01:23:59.0838] [0] End of Setup
Don't mind, I fixed it , because of lock exe or something... I just have to extract and run CU7 from local server.


April 5th, 2015 1:48am

Hi all,

After upgraded EX01 to CU7, I try to upgrade EX02 too but error happens when I run commands

setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms setup.exe /mode:upgrade /IAcceptExchangeServerLicenseTerms

Welcome to Microsoft Exchange Server 2013 Cumulative Update 7 Unattended Setup
Copying Files...
File copy complete. Setup will now collect additional information needed for
installation.

The Exchange Server setup operation didn't complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

Here content of ExchangeSetupLogs\ExchangeSetup.log

[04/05/2015 01:23:55.0704] [0] Starting Microsoft Exchange Server 2013 Cumulative Update 7 Setup
[04/05/2015 01:23:55.0704] [0] **********************************************
[04/05/2015 01:23:55.0704] [0] Local time zone: (UTC+07:00) Bangkok, Hanoi, Jakarta.
[04/05/2015 01:23:55.0704] [0] Operating system version: Microsoft Windows NT 6.1.7601 Service Pack 1.
[04/05/2015 01:23:55.0719] [0] Setup version: 15.0.1044.25.
[04/05/2015 01:23:55.0719] [0] Logged on user: mydomain\administrator.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='mode', Value='Upgrade'.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='iacceptexchangeserverlicenseterms', Value=''.
[04/05/2015 01:23:55.0782] [0] Command Line Parameter Name='sourcedir', Value='D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] RuntimeAssembly was started with the following command: '/mode:upgrade /IAcceptExchangeServerLicenseTerms /sourcedir:D:\Upgrade\Exchange2013-x64-cu7'.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\v8.0, wasn't found.
[04/05/2015 01:23:56.0765] [0] The registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14, wasn't found.
[04/05/2015 01:23:56.0780] [0] Copying Files...
[04/05/2015 01:23:56.0796] [0] Starting copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] Finished copy from D:\Upgrade\Exchange2013-x64-cu7\Setup\ServerRoles\Common to C:\Windows\Temp\ExchangeSetup.
[04/05/2015 01:23:59.0760] [0] File copy complete. Setup will now collect additional information needed for installation.
[04/05/2015 01:23:59.0775] [0] Assembly dll file location is C:\Windows\Temp\ExchangeSetup\Microsoft.Exchange.Setup.Console.dll
[04/05/2015 01:23:59.0838] [0] [ERROR] An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
[04/05/2015 01:23:59.0838] [0] CurrentResult SetupLauncherHelper.loadassembly:444: 1
[04/05/2015 01:23:59.0838] [0] The Exchange Server setup operation didn't complete.  More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
[04/05/2015 01:23:59.0838] [0] CurrentResult main.run:235: 1
[04/05/2015 01:23:59.0838] [0] CurrentResult setupbase.maincore:396: 1
[04/05/2015 01:23:59.0838] [0] End of Setup
Don't mind, I fixed it , because of lock exe or something... I just have to extract and run CU7 from local server.


Free Windows Admin Tool Kit Click here and download it now
April 5th, 2015 1:48am

First thing, you only need to run the Schema and AD updates once per organization, so once you ran it on the first server, there is no need to run it on the second server.  

Next, I;m going to assume the DB is active on the second server?  If that's true on the first server, let's see what the status of the DB copy, run:

Get-MailboxDatabaseCopyStatus 

and see if that says the database is healthy.

April 6th, 2015 9:10am

Yes, I ran Schema and AD updates on EX01 first, so it's not necessary to run on EX02 again. Because of lock exe or something... I just have to extract and run CU7 from local EX02 server. I upgraded EX02 to CU7 successfully.

After upgrade to CU7, I see less error in Event Viewer than before, but some events 2153 still appear on EX02:

Log Name:      Application
Source:        MSExchangeRepl
Date:          4/7/2015 10:24:43 AM
Event ID:      2153
Task Category: Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX02.mydomain.com
Description:
The log copier was unable to communicate with server 'EX01.mydomain.com'. The copy of database 'Mailbox Database 01\EX02' is in a disconnected state. The communication error was: An error occurred while communicating with server 'EX01'. Error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. The copier will automatically retry after a short delay.

Log Name:      Application
Source:        MSExchangeRepl
Date:          4/7/2015 10:45:56 AM
Event ID:      2153
Task Category: Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX02.mydomain.com
Description:
The log copier was unable to communicate with server 'EX01.mydomain.com'. The copy of database 'Mailbox Database 01\EX02' is in a disconnected state. The communication error was: An error occurred while communicating with server 'EX01'. Error: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. The copier will automatically retry after a short delay.
Will keep my eyes on them.

I have a question, I want to check "Mailbox Database 01" edb file on EX01 & EX02. Should I take one of them into maintenance mode, stop all Exchange service then check with eseutil /mh command ?

Free Windows Admin Tool Kit Click here and download it now
April 7th, 2015 12:35am

if the edb file is mounted it should be good.
April 7th, 2015 8:44am

Hi Jack,

Lots of things going on as I see.

Back to where I left, correcting the Witness Server might solve your database dismounting issue.

However I still feel like your cluster is not set right.
Why DynamicWeight,NodeWeight are empty?
Get-Cluster | fl *Wit* is missing WitnessDynamicWeight

Read though if you like on Dynamic Quorum

To Check if its enabled(1) or not:

(Get-Cluster).DynamicQuorum
Free Windows Admin Tool Kit Click here and download it now
April 7th, 2015 11:45am

Hi Satyajit,
As result of Get-ClusterNode and Get-Cluster : DynamicWeight,NodeWeight are empty , so I guess my DAG/cluster is not set right lead to "lost quorum" problem sometimes ?
I read link about Dynamic Quorum, thank you for that link, but my Exchange servers OS is Windows Server 2008 R2 SP1 (sorry because didn't mention it before).
I didn't notice about Quorum things before, after created DAG01 , add 2 member EX01, EX02 , configure mailbox database copy I just leave it.

Command (Get-Cluster).DynamicQuorum don't show any result.
I read Configure and Manage the Quorum in a Windows Server 2012 Failover Cluster , I used "Failover Cluster Manager" to configure Cluster Quorum Settings again (File Share Witness) , but (Get-Cluster).DynamicQuorum result empty.

April 7th, 2015 11:59pm

How often did the process crashes happen prior to the update? 
Free Windows Admin Tool Kit Click here and download it now
April 8th, 2015 9:23am

It crashed about once per week. My users experience "service down" because their MS Outlook cannot connect to Exchange, I check event log and see events:

7031 - "The Microsoft Exchange RPC Client Access service terminated unexpectedly" or "The Microsoft Exchange Mailbox Assistants service terminated unexpectedly" on both EX01 & EX02

2153 "The log copier was unable to communicate with server 'EX01.mydomain.com'. The copy of database 'Mailbox Database 01\EX02' is in a disconnected state. The communication error was: An error occurred while communicating with server 'EX01'. Error: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. The copier will automatically retry after a short delay" on EX02

After upgrade to CU7, errors still appear in Event Viewer but it doesn't crash until now.

April 8th, 2015 10:58pm

Hi Jack,

Dynamic Quorum is Windows Server 2012 only feature. :)

Guess need to rethink things now.

Free Windows Admin Tool Kit Click here and download it now
April 10th, 2015 6:12am

It crashed about once per week. My users experience "service down" because their MS Outlook cannot connect to Exchange, I check event log and see events:

7031 - "The Microsoft Exchange RPC Client Access service terminated unexpectedly" or "The Microsoft Exchange Mailbox Assistants service terminated unexpectedly" on both EX01 & EX02

2153 "The log copier was unable to communicate with server 'EX01.mydomain.com'. The copy of database 'Mailbox Database 01\EX02' is in a disconnected state. The communication error was: An error occurred while communicating with server 'EX01'. Error: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. The copier will automatically retry after a short delay" on EX02

After upgrade to CU7, errors still appear in Event Viewer but it doesn't crash until now.

Which errors? The RPC Service/Mailbox Assistance Service crashing? or the communication one?
April 10th, 2015 8:58am

When my user experience "service down" I check event logs and found :

Event 7031 "The RPC Service/Mailbox Assistance Service crashing" on both EX01 & EX02

Event 2153 on EX02

"Mailbox Database 01" is mount on EX01 , standby on EX02

I upgraded to CU7 on 29 March 2015, since that day my users haven't experience "service down" .

Event 7031 still appear on EX01 until 5 April 2015
Event 7031 still appear on EX02 until 6 April 2015
Since 6 April 2015 there is no 7031 event on both servers anymore
Event 7031 is not only related to RPC Service or Mailbox Assistance Service but also "The Microsoft Exchange Transport " , "The Microsoft Exchange Frontend Transport ", "The Microsoft Exchange Health Manager ", "The Microsoft Exchange Mailbox Transport Delivery"

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:37:47 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Mailbox Transport Delivery service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:23:57 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Health Manager service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:12:53 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Frontend Transport service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Event 2153 still appear on EX02 til now, as I mention above "Mailbox Database 01" is mount on EX01 , standby on EX02.

Free Windows Admin Tool Kit Click here and download it now
April 13th, 2015 12:21am

When my user experience "service down" I check event logs and found :

Event 7031 "The RPC Service/Mailbox Assistance Service crashing" on both EX01 & EX02

Event 2153 on EX02

"Mailbox Database 01" is mount on EX01 , standby on EX02

I upgraded memory (increase RAM) on both servers on 29 March 2015.
I changed witness server on DAG to new server on 03 April 2015.
I upgraded to CU7 on 04 March 2015, since that day my users haven't experience "service down" .

Event 7031 still appear on EX01 until 5 April 2015
Event 7031 still appear on EX02 until 6 April 2015
Since 6 April 2015 there is no 7031 event on both servers anymore
Event 7031 is not only related to RPC Service or Mailbox Assistance Service but also "The Microsoft Exchange Transport " , "The Microsoft Exchange Frontend Transport ", "The Microsoft Exchange Health Manager ", "The Microsoft Exchange Mailbox Transport Delivery"

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:37:47 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Mailbox Transport Delivery service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:23:57 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Health Manager service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:12:53 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Frontend Transport service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Event 2153 still appear on EX02 til now, as I mention above "Mailbox Database 01" is mount on EX01 , standby on EX02.


April 13th, 2015 4:19am

When my user experience "service down" I check event logs and found :

Event 7031 "The RPC Service/Mailbox Assistance Service crashing" on both EX01 & EX02

Event 2153 on EX02

"Mailbox Database 01" is mount on EX01 , standby on EX02

I upgraded memory (increase RAM) on both servers on 29 March 2015.
I changed witness server on DAG to new server on 03 April 2015.
I upgraded to CU7 on 04 March 2015, since that day my users haven't experience "service down" .

Event 7031 still appear on EX01 until 5 April 2015
Event 7031 still appear on EX02 until 6 April 2015
Since 6 April 2015 there is no 7031 event on both servers anymore
Event 7031 is not only related to RPC Service or Mailbox Assistance Service but also "The Microsoft Exchange Transport " , "The Microsoft Exchange Frontend Transport ", "The Microsoft Exchange Health Manager ", "The Microsoft Exchange Mailbox Transport Delivery"

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:37:47 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Mailbox Transport Delivery service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:23:57 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Health Manager service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:12:53 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Frontend Transport service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Event 2153 still appear on EX02 til now, as I mention above "Mailbox Database 01" is mount on EX01 , standby on EX02.


  • Edited by Jack Chuong Monday, April 13, 2015 10:38 AM
Free Windows Admin Tool Kit Click here and download it now
April 13th, 2015 4:19am

When my user experience "service down" I check event logs and found :

Event 7031 "The RPC Service/Mailbox Assistance Service crashing" on both EX01 & EX02

Event 2153 on EX02

"Mailbox Database 01" is mount on EX01 , standby on EX02

I upgraded memory (increase RAM) on both servers on 29 March 2015.
I changed witness server on DAG to new server on 03 April 2015.
I upgraded to CU7 on 04 March 2015, since that day my users haven't experience "service down" .

Event 7031 still appear on EX01 until 5 April 2015
Event 7031 still appear on EX02 until 6 April 2015
Since 6 April 2015 there is no 7031 event on both servers anymore
Event 7031 is not only related to RPC Service or Mailbox Assistance Service but also "The Microsoft Exchange Transport " , "The Microsoft Exchange Frontend Transport ", "The Microsoft Exchange Health Manager ", "The Microsoft Exchange Mailbox Transport Delivery"

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:37:47 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Mailbox Transport Delivery service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:23:57 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Health Manager service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Log Name:      System
Source:        Service Control Manager
Date:          4/5/2015 7:12:53 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX01.mydomain.com
Description:
The Microsoft Exchange Frontend Transport service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Event 2153 still appear on EX02 til now, as I mention above "Mailbox Database 01" is mount on EX01 , standby on EX02.


  • Edited by Jack Chuong Monday, April 13, 2015 10:38 AM
April 13th, 2015 4:19am

As dynamic quorum is not there you need to have n/2+1 = 2/2+1=2 votes always to keep the cluster running.

Hence atleast 1Witness+1Node or both 2 Nodes need to be alive to keep your DBs mounted.

So the Witness fix, should keep it on, till you solve the service crashing issue with Hinte.

Free Windows Admin Tool Kit Click here and download it now
April 13th, 2015 5:40am

Thank you very much Satyajit,

It seems because something wrong with my old witness server so I don't have enough vote to mount DB when problem happen and change witness server to new server fixed the problem.

When I upgrade RAM on 29 March, I put EX02 to maintenance mode but when I shutdown EX02, "Mailbox Database 01" cannot be mounted on EX01, my users experience "service down" until I start up EX02 . Same thing happened with EX02

When I upgrade CU7 on 04 April (after changed witness server on 03 April) everything worked right, after put EX02 to maintenance mode, I run setup.exe on EX02 (Exchange services on EX02 is stopped automatically), "Mailbox Database 01" can be mounted on EX01.

Should I concern about event 2153 still appears on EX02 ?

April 13th, 2015 6:48am

Hi Jack,

If your error's are not that frequent and DAG failovers works fine, no potential user impact. You might choose not to action on it for now.

However these errors would indicate network issues. Run the below cmdlet to check, if your DAG network looks okay.

Get-DatabaseAvailabilityGroupNetwork

More-over if the error is only on EX02, you might want to re-seed the database on EX02 afresh.

Suspend-MailboxDatabaseCopy -Identity DB1\EX02 
Update-MailboxDatabaseCopy -Identity DB1\EX02 
Resume-MailboxDatabaseCopy -Identity DB1\EX02

Also check, if the error is coming around specific time only (during backup jobs running etc.)

How is the IOPS > 0.07*'total_users on the active DB' for your Exchange storage.

I/O Database Reads Average Latency (msec) <20ms

I/O Database Writes Average Latency (msec) <10ms

References:

The log copier was unable to communicate -Exchange2010

Event ID 2153 is logged

Exchange 2013 Databases  shift automatically between DAG members on daily basis

Free Windows Admin Tool Kit Click here and download it now
April 13th, 2015 8:53am


Get-DatabaseAvailabilityGroupNetwork

Identity           ReplicationEnabled              Subnets

DAG01\MapiDagNetwork   True          {{192.168.2.0/24,Up}}
DAG01\ReplicationDagNetwork01 True    {{10.10.10.0/24,Up}}

I should explain this. At beginning each Exchange server have 2 physical network interfaces : Lan1 & Lan2 (1Gbps) , Lan1 is plugged into switch port (100 Mbps) : DAG01\MapiNetwork .
Then I followed article here : http://jerridwills.com/2013/02/04/creating-a-two-node-exchange-2013-dag/
to add more Replication network. Lan2 on EX01 is plugged into Lan2 on EX02 directly by a network cable .
I verified that Mapi traffic go through DAG01\MapiNetwork and Replication traffic (port 64327) go through DAG01\ReplicationDagNetwork01 by watching Wireshark .

I have 500+ mailbox users , 3 mailbox databases : 01 , 02 , 03
Mailbox Database 01 : mount on EX01, standby on EX02, store about 400 mailboxes.
Mailbox Database 02 : mount on EX02, standby on EX01, store about 80 mailboxes.
Mailbox Database 03 : mount on EX02, standby on EX01, store about 20 mailboxes.

Get-MailboxDatabase "Mailbox Database 01" -Status | FL AvailableNewMailboxSpace, DatabaseSize

AvailableNewMailboxSpace : 232.2 GB (249,319,587,840 bytes)
DatabaseSize             : 594.1 GB (637,952,655,360 bytes)

Get-MailboxDatabase "Mailbox Database 03" -Status | FL AvailableNewMailboxSpace, DatabaseSize

AvailableNewMailboxSpace : 10.85 GB (11,651,973,120 bytes)
DatabaseSize             : 21.76 GB (23,362,863,104 bytes)

Get-MailboxDatabase "Mailbox Database 02" -Status | FL AvailableNewMailboxSpace, DatabaseSize

AvailableNewMailboxSpace : 79.99 GB (85,883,387,904 bytes)
DatabaseSize             : 147.4 GB (158,251,155,456 bytes)

Because of some objective reasons I cannot balance number mailboxes in each mailbox database now. Some information about DAG network configuration:

Get-DatabaseAvailabilityGroupNetwork | fl

RunspaceId         : 9b3c6459-6a27-41c5-8398-01ea83384597
Name               : MapiDagNetwork
Description        :
Subnets            : {{192.168.2.0/24,Up}}
Interfaces         : {{EX01,Up,192.168.2.3}, {EX02,Up,192.168.2.4}}
MapiAccessEnabled  : True
ReplicationEnabled : True
IgnoreNetwork      : False
Identity           : DAG01\MapiDagNetwork
IsValid            : True
ObjectState        : New

RunspaceId         : 9b3c6459-6a27-41c5-8398-01ea83384597
Name               : ReplicationDagNetwork01
Description        :
Subnets            : {{10.10.10.0/24,Up}}
Interfaces         : {{EX01,Up,10.10.10.3}, {EX02,Up,10.10.10.4}}
MapiAccessEnabled  : False
ReplicationEnabled : True
IgnoreNetwork      : False
Identity           : DAG01\ReplicationDagNetwork01
IsValid            : True
ObjectState        : New

Get-MailboxDatabaseCopyStatus -Server ex01 | fl name,*incoming*,*outgoing*


Name                      : Mailbox Database 01\EX01
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : Mailbox Database 02\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : Mailbox Database 03\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections       :

Get-MailboxDatabaseCopyStatus -Server ex02 | fl name,*incoming*,*outgoing*

Name                      : Mailbox Database 02\EX02
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : Mailbox Database 01\EX02
IncomingLogCopyingNetwork : {EX01,ReplicationDagNetwork01,An error occurred while communicating with server
                            'EX01'. Error: Unable to write data to the transport connection: An established
                            connection was aborted by the software in your host machine.}
OutgoingConnections       :

Name                      : Mailbox Database 03\EX02
IncomingLogCopyingNetwork :
OutgoingConnections       : {}
I can disable replication traffic on MapiDagNetwork but I don't think it is problem.
Something wrong with IncomingLogCopyingNetwork for Mailbox Database 01\EX02 , event 2153 happen with "Mailbox Database 01" on EX02 only, I think this is reason.
Error coming around specific time only : working time (from 7 AM to 5 PM) several times per day
My databases maintenance time : 1:00 AM - 5:00 AM daily
I will try re-seed Mailbox Database 01 on EX02 on weekend

I read an article about IOPS here : http://blog.synology.com/?p=2086 , I will try IOPS asap.
Each my server have 3 physical disks, raid 5 ==> 2 logical disks : C & D
Exchange 2013 is installed on C drive (configuration , binary , CAS logs)
Mailbox Databases are stored in D drive (edb files, MB logs)
What data should I collect ? (C or D or total ?)

April 14th, 2015 12:43am

Hi Jack,

In your case you should monitor both C & D as RAID config would strip the data across all 3 disks.

Don't worry much about the below recommendations if you get the IOPS and Latency right.

As per recommendations:

OS\Exchange disk and Database\Logs disks should be seperate.(2 seperate volumes\disks)

Below are the RAID configs:

Operating System, System, or Pagefile Volume:  RAID-1 (or RAID-1/0)

Separated Mailbox Database and Log Volumes: Mailbox volumes is RAID-1/0 ; For log volumes, RAID-1 (or RAID-1/0.)

When using RAID-5 configurations for the operating system, pagefile, or Exchange data volumes, note the following:

  • You should have no more than 7 disks per array group and array controller high-priority scrubbing and surface scanning enabled.

In regrads to your network error:

NOTE:-

In the event of a failure affecting the Replication network, if the MAPI network is unaffected by the failure, log shipping and seeding operations will revert to use the MAPI network, even if the MAPI network has it's ReplicationEnabled property set to False.

You error indicates issue with the replication network.

"Name                      : Mailbox Database 01\EX02
IncomingLogCopyingNetwork : {EX01,ReplicationDagNetwork01,"

Never seen anyone doing this: "Lan2 on EX01 is plugged into Lan2 on EX02 directly by a network cable ." (Great to see that it works. :) )

1. Try re-routing the connection via a switch (but 100Mbps seems to be low)

2. Try this Workaround to isolate the issue:

Exchange 2010 DAG replication issues -Replication Network issue:

http://blogs.technet.com/b/samdrey/archive/2014/09/10/exchange-2010-dag-replication-issue-msexchangerepl-2153-potentially-an-error-while-trying-to-update-databasecopy.aspx

Heads up on the network issue troubleshooting: Exchange 2010 / 2013: What constitutes a failure of the replication network

April 15th, 2015 12:28am

Hi Sayajit,
Because of cost issue, I have to use 100 Mbps switch port but 100 Mbps connection is not enough for Mapi + Replication traffic so I used another network interface for replication traffic, and to avoid configure new VLAN on switch + take advantage of 1 Gbps network cards ==> I connected them directly by a cable :)
Route table on my servers seem fine

I'm running performance tool to collect data for IOPS in next 24 hours.

Thank you for your links, but with "Exchange 2010 DAG replication issues -Replication Network issue" , SammyKrosoft 's problem is "Error: Unable to read data to the transport connection" and mine is "Error: Unable to write data to the transport connection".
I can Suspend , Update , Resume ==> reseed Mailbox Database 01 on EX02 by using ECP before, but I think I will do this with command for sure if I can see error like SammyKrosoft said.

I will also try "workaround":

  1. Disable replication on ReplicationDagNetwork01 + Restart MSExchange Replication service
  2. Check result of Get-MailboxDatabaseCopyStatus -Server ex01 | fl name,*incoming*,*outgoing* if there still error
  3. Enable replication on ReplicationDagNetwork01 + Restart MSExchange Replication service
  4. Check again.
April 15th, 2015 8:44am

Last night I tried reseed Mailbox Database 01\EX02 , about 8:15 PM
I checked event viewer, last 2153 event happened on 7:49 PM, check around :

Get-MailboxDatabaseCopyStatus -Server ex02 | fl name,*incoming*,*outgoing*

Name                      : Mailbox Database 07\EX02
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : Mailbox Database 01\EX02
IncomingLogCopyingNetwork : {EX01,ReplicationDagNetwork01,An error occurred while communicating with server
                            'EX01'. Error: Unable to write data to the transport connection: An established
                            connection was aborted by the software in your host machine.}
OutgoingConnections       :

Name                      : Mailbox Database 99\EX02
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Get-MailboxDatabaseCopyStatus -Server ex01 | fl name,*incoming*,*outgoing*

Name                      : Mailbox Database 01\EX01
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : Mailbox Database 99\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : Mailbox Database 07\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections       :

Then I suspend and update Mailbox Database 01\EX02

Suspend-MailboxDatabaseCopy -Identity "Mailbox Database 01\EX02"

Update-MailboxDatabaseCopy -Identity "Mailbox Database 01\EX02" -Network "DAG01\ReplicationDagNetwork01"

Confirm
Are you sure you want to perform this action?
Seeding database copy "Mailbox Database 01\EX02".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"):
The seeding operation failed. Error: An error occurred while running prerequisite checks. Error: Log files exist in
'D:\Exchange Mailbox Database\Mailbox Database 01'. You must remove them before database seeding or reseeding can be
performed. You can use the Update-MailboxDatabaseCopy cmdlet with the -DeleteExistingFiles or -SafeDeleteExistingFiles
parameter to do this. [Database: Mailbox Database 01, Server: EX02.mydomain.com]
    + CategoryInfo          : InvalidOperation: (Mailbox Database 01:String) [Update-MailboxDatabaseCopy], SeedPrepare
   Exception
    + FullyQualifiedErrorId : [Server=EX02,RequestId=56f71c07-6209-4aae-94cb-bf3cfc0a38fd,TimeStamp=4/22/2015 1:0
   5:56 PM] [FailureCategory=Cmdlet-SeedPrepareException] 5F5B4C53,Microsoft.Exchange.Management.SystemConfigurationT
  asks.UpdateDatabaseCopy
    + PSComputerName        : ex02.mydomain.com

I want they use DAG01\ReplicationDagNetwork01 (1Gbps) for reseeding database, then I tried to add -SafeDeleteExistingFiles

Update-MailboxDatabaseCopy -Identity "Mailbox Database 01\EX02" -Network "DAG01\ReplicationDagNetwork01" -SafeDeleteExistingFiles

Confirm
Are you sure you want to perform this action?
Seeding database copy "Mailbox Database 01\EX02".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"):
The seeding operation failed. Error: An error occurred while running prerequisite checks. Error: An error occurred
when trying to run the database redundancy validation checks for database 'Mailbox Database 01' due to
'SafeDeleteExistingFiles' being specified. Error: There were database redundancy check failures for database 'Mailbox
Database 01' that may be lowering its redundancy and putting the database at risk of data loss. Redundancy Count: 1.
Expected Redundancy Count: 2. Detailed error(s):
        ex02:
        Database 'Mailbox Database 01' does not have enough copies configured to meet the validation criteria.
         [Database: Mailbox Database 01, Server: EX02.mydomain.com]
    + CategoryInfo          : InvalidOperation: (Mailbox Database 01:String) [Update-MailboxDatabaseCopy], SeedPrepare
   Exception
    + FullyQualifiedErrorId : [Server=EX02,RequestId=48374aa6-1198-4976-a435-3eb034dc5b58,TimeStamp=4/22/2015 1:0
   8:44 PM] [FailureCategory=Cmdlet-SeedPrepareException] 458ED9A2,Microsoft.Exchange.Management.SystemConfigurationT
  asks.UpdateDatabaseCopy
    + PSComputerName        : ex02.mydomain.com

I think I should stop here, search more about -DeleteExistingFiles before continue. So I Resume-MailboxDatabaseCopy "Mailbox Database 01\EX02"

Everything is fine, I checked again

[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus -Server ex02 | fl name,*incoming*,*outgoing*


Name                      : Mailbox Database 07\EX02
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : Mailbox Database 01\EX02
IncomingLogCopyingNetwork : {EX01,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : Mailbox Database 99\EX02
IncomingLogCopyingNetwork :
OutgoingConnections       : {}



[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus -Server ex01 | fl name,*incoming*,*outgoing*


Name                      : Mailbox Database 01\EX01
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : Mailbox Database 99\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : Mailbox Database 07\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections

What is going on :) ? IncomingLogCopyingNetwork for Mailbox Database 01\EX02 is fine :) til mid night

This morning I view events on EX02 and event 2153 appears again on 7:30 AM , I run command et-MailboxDatabaseCopyStatus again

Get-MailboxDatabaseCopyStatus -Server ex02 | fl name,*incoming*,*outgoing*

Name                      : Mailbox Database 07\EX02
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : Mailbox Database 01\EX02
IncomingLogCopyingNetwork : {EX01,ReplicationDagNetwork01,An error occurred while communicating with server
                            'EX01'. Error: Unable to write data to the transport connection: An established
                            connection was aborted by the software in your host machine.}
OutgoingConnections       :

Name                      : Mailbox Database 99\EX02
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Get-MailboxDatabaseCopyStatus -Server ex01 | fl name,*incoming*,*outgoing*

Name                      : Mailbox Database 01\EX01
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : Mailbox Database 99\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : Mailbox Database 07\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections       :

Any suggest ?

Free Windows Admin Tool Kit Click here and download it now
April 22nd, 2015 11:41pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics