2012 R2 with exchange 2013 SP1 2 member servers and 1 DAG. One of the exchange servers lost 2 drives on its RAID 5 array and i need to see if i can recover the VHDX. So email still flows, i need to move the witness server to a different server. Is it safe to move to a different witness server when 1 member server is offline?

2012 R2 with exchange 2013 SP1 2 member servers and 1 DAG. One of the exchange servers lost 2 drives on its RAID 5 array and i need to see if i can recover the VHDX. So email still flows, i need to move the witness server to a different server. Is it safe to move to a different witness server when 1 member server is offline?

Move how? With Vmotion ( or whatever they call it in Windows)? If the FSW remains up, quorum will be maintained since there are 2 votes ( The FSW and the running Mailbox Server) If you down the FSW, quorum will be lost and the databases will dismount.

2012 R2 with exchange 2013 SP1 2 member servers and 1 DAG. One of the exchange servers lost 2 drives on its RAID 5 array and i need to see if i can recover the VHDX. So email still flows, i need to move the witness server to a different server. Is it safe to move to a different witness server when 1 member server is offline?

Move how? With Vmotion ( or whatever they call it in Windows)? If the FSW remains up, quorum will be maintained since there are 2 votes ( The FSW and the running Mailbox Server) If you down the FSW, quorum will be lost and the databases will dismount.

Not sure what Vmotion is. I was simply thinking about copying the existing witness directory over to another server (after creating the folder and assigning permissions). Then in ECP, point the DAG to the new witness server. Or create the witness folder on the new server, assigning permissions and then in ECP point the DAG to the new witness server

2012 R2 with exchange 2013 SP1 2 member servers and 1 DAG. One of the exchange servers lost 2 drives on its RAID 5 array and i need to see if i can recover the VHDX. So email still flows, i need to move the witness server to a different server. Is it safe to move to a different witness server when 1 member server is offline?

Move how? With Vmotion ( or whatever they call it in Windows)? If the FSW remains up, quorum will be maintained since there are 2 votes ( The FSW and the running Mailbox Server) If you down the FSW, quorum will be lost and the databases will dismount.

Not sure what Vmotion is. I was simply thinking about copying the existing witness directory over to another server (after creating the folder and assigning permissions). Then in ECP, point the DAG to the new witness server. Or create the witness folder on the new server, assigning permissions and then in ECP point the DAG to the new witness server

Ok, I assumed you meant this was a virtual guest. Why the need to change the FSW in this case?

As Ed mentioned, its a pretty straightforward process, but I have never done that with the DAG at the point where the FSW is required - which appears to be where you are at.

Ok, I assumed you meant this was a virtual guest. Why the need to change the FSW in this case?

As Ed mentioned, its a pretty straightforward process, but I have never done that with the DAG at the point where the FSW is required - which appears to be where you are at.

The current witness IS on a VM on the offline server. So when i boot off a CD to run recovery software, the witness will be down and all email will stop flowing. So i need to move to a different witness server. If this does not work, i should be able to return to the current witness server and keep mail flowing, yes?

You don't copy anything over, just run the cmdlet.  But hold your breath because databases may dismount, so you should schedule it when the fewest number of people will scream.

Before i run the cmdlet, make the folder and assign permissions or will permissions happen when the cmdlet is run? I want to mention, i want to move the witness to an domain controller. Is that a problem?

• Edited by forgiven 5 hours 29 minutes ago

I wouldn't do that.  The computer must have Exchange Trusted Subsystem as a member of its local Administrators group.  Domain controllers don't have a local Administrators group, so you must make that group a member of the domain's Administrators group.

So, i ran the cmdlet and am asked Identity. Is that the identity of the mailbox DB or of the DAG?

After looking at that cmdlet, it clearly is asking for the identity of the DAG. So, i typed in the DAG identity and received:

Please post the text and not a picture.  It's really hard to read those pictures.

Well, that's saying that you can't do it without both DAG members being alive.

You do have the options of forcing quorum or removing the dead DAG member from the DAG.

You don't copy anything over, just run the cmdlet.  But hold your breath because databases may dismount, so you should schedule it when the fewest number of people will scream.

Before i run the cmdlet, make the folder and assign permissions or will permissions happen when the cmdlet is run? I want to mention, i want to move the witness to an domain controller. Is that a problem?

• Edited by forgiven Wednesday, July 08, 2015 1:58 AM

You don't copy anything over, just run the cmdlet.  But hold your breath because databases may dismount, so you should schedule it when the fewest number of people will scream.

Before i run the cmdlet, make the folder and assign permissions or will permissions happen when the cmdlet is run? I want to mention, i want to move the witness to an domain controller. Is that a problem?

• Edited by forgiven Wednesday, July 08, 2015 1:58 AM

You don't copy anything over, just run the cmdlet.  But hold your breath because databases may dismount, so you should schedule it when the fewest number of people will scream.

Before i run the cmdlet, make the folder and assign permissions or will permissions happen when the cmdlet is run? I want to mention, i want to move the witness to an domain controller. Is that a problem?

• Edited by forgiven Wednesday, July 08, 2015 1:58 AM

Please post the text and not a picture.  It's really hard to read those pictures.

Well, that's saying that you can't do it without both DAG members being alive.

You do have the options of forcing quorum or removing the dead DAG member from the DAG.

It doesn't look positive about recovering the defunct array which the exchange DB residing on and becaause of that, the exchange VM will not start. So i am wondering about this:

After rebuilding and initializing the array, make a new VM for exchange. Update server 2012 r2, Install exchange and upadate exchange. Add new server to the DAG and reseed the good copy? What would you do if you were in this spot?

Based on the limited information I have about your situation

I think I would force quorum, remove the database copies from the bad VM and remove the server from the DAG, build a new VM, reinstall Exchange with /RecoverServer, add it to the DAG and reseed the copies.

Have you tried?

No and i dont want to "try" anything on a crippled DAG that i dont know or not if it will work or make things worse. That's what forums like these are for, you know, to ask questions and hopefully get answers.

Then I recommend you hire an experienced consultant to hold your hand, or open a ticket with Microsoft Support.

Good luck.

Ed, you have been extremely helpful and i thank you for your help.

So to force quorum, my research says to run: net start clussvc /fq

https://technet.microsoft.com/en-us/library/dd197500%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396

This will effectively remove the quorum and the surviving member will operate as standalone, correct? I'm not looking for someone to hold my hand but i do want to make sure i am doing the right thing since i do not see forcing quorum in any article or tech article.

You should use the native Exchange cmdlets.

Let's recap.  Do you have mounted databases now?

Just remove the failed server from the DAG, which will leave you with a DAG that has one member.  No FSW is required at that point.  All your databases should mount if they're not mounted already.  Then you should be able to begin restorative efforts to get a second copy again.

Okay, what I said should lead you down the road to recovery.  Before removing the server from the DAG, you may have to remove the database copies.  No matter, they're gone anyway.

I ran 

Remove-DatabaseAvailabilityGroupServer -Identity DAG3 -MailboxServer MBX1

Enter:

Get-MailboxDatabaseCopyStatus -Server MBXserver

The names in the left-hand column are your database copies to remove.

That worked. I was entering the DAG name not the DB name.

So then i ran 

Remove-DatabaseAvailabilityGroupServer -Identity DAG3 -MailboxServer MBX1

WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file
"C:\ExchangeSetupLogs\DagTasks\dagtask_2015-07-08_23-54-47.342_remove-databaseavailabiltygroupserver.log".
There was a problem changing the quorum model for database availability group DAGDB1. Error: An error occurred while
attempting a cluster operation. Error: Cluster API failed: "SetClusterQuorumResource() failed with 0x1725. Error: A
quorum of cluster nodes was not present to form a cluster"

Looking in the EAC, the witness server has been changed to a different server (even though the command reported an error when run). So i changed it back to the "old" witness server using the above cmdlet and reran

Remove-DatabaseAvailabilityGroupServer -Identity DAG3 -MailboxServer MBX1

Received the same error. Here is the last bit of the referenced log file:

Successfully resolved the servers based on the stopped servers list.
[2015-07-08T23:58:17] The following servers are in the StartedServers list (The list is the StartedServers property of the DAG in AD):
[2015-07-08T23:58:17] The following servers are in the StoppedServers list:
[2015-07-08T23:58:17] Verifiying that the members of database availability group 'DAGDB1' are also members of the cluster.
[2015-07-08T23:58:17] Verifying that the members of cluster 'DAGDB1' are also members of the database availability group.
[2015-07-08T23:58:17] There are 2 nodes in the cluster.
[2015-07-08T23:58:17] DAG DAGDB1 has 2 servers:
[2015-07-08T23:58:17] DAG DAGDB1 contains server servername.
[2015-07-08T23:58:17] DAG DAGDB1 contains server servername.
[2015-07-08T23:58:17] Reopening a handle to the cluster using the names [servername].
[2015-07-08T23:58:17] CheckClusterStateForDagServerRemoval left. m_removeNode=True, m_destroyCluster=False.
[2015-07-08T23:58:17] Checking if msexchangerepl is running on <online server>.
[2015-07-08T23:58:17] The CNO is currently Online.
[2015-07-08T23:58:19] Updated Progress 'Removing server 'offline server' from 'DAGDB1'.' 2%.
[2015-07-08T23:58:19] Working
[2015-07-08T23:58:19] The core cluster group 'Cluster Group' is currently on machine 'online server'.
[2015-07-08T23:58:19] ShouldBeFileShareWitness has been called anticipating a node removal. Decrementing the server count from 2.
[2015-07-08T23:58:19] There is a single server in the cluster, and StoppedMailboxServers is empty. No file share witness needed.
[2015-07-08T23:58:19] SkipDagValidation = False; IsQuorumTypeFSW = True; ShouldbeFSW = False
[2015-07-08T23:58:19] IsDagFailedOverToOtherSite: The DAG is in DAC mode of 'off'.
[2015-07-08T23:58:19] Updated Progress 'No longer using file share witness share because there are an odd number of members in the database availability group.' 4%.
[2015-07-08T23:58:19] Working
[2015-07-08T23:58:19] Setting cluster quorum to MNS
[2015-07-08T23:58:19] Setting cluster quorum resource to the netname resource (i.e. MNS quorum).
[2015-07-08T23:58:19] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2015-07-08_23-58-16.998_remove-databaseavailabiltygroupserver.log".
[2015-07-08T23:58:19] WriteError! Exception = Microsoft.Exchange.Management.Tasks.DagTaskProblemChangingQuorumException: There was a problem changing the quorum model for database availability group DAGDB1. Error: An error occurred while attempting a cluster operation. Error: Cluster API failed: "SetClusterQuorumResource() failed with 0x1725. Error: A quorum of cluster nodes was not present to form a cluster"
[2015-07-08T23:58:19] Updated Progress 'Done!' 100%.
[2015-07-08T23:58:19] COMPLETED
remove-databaseavailabiltygroupserver explicitly called CloseTempLogFile().

Try running the command with -SkipDagValidation

I hope that works out for you because it's a better solution.

The array rebuild did not work (i do have 1 drive left with the working array config but am ready to move past this).

So i just want to make sure i am on the same page. The server that went offline has 2 arrays, 1 for c drive and 1 for d drive. Exchange is installed on the c drive. D drive housed the DB only. The d drive is the drive that was lost. So when i go to run /recoverserver, will the installed exchange conflict with /recoverserver?

• Edited by forgiven 19 hours 20 minutes ago

Not surprised since 2 drives were lost, RAID5 cannot tolerate that as there is one parity. I'm not advocating RAID 6 either BTW....

If the OS for the Exchange server on that VM host is OK, and the Exchange binaries are also on the C:\ drive then you do not need to \recoverserver.  Exchange is already running.  You need to fix the database copies.  So get the storage squared away with the same drive letters etc. as before.

Then you can re-add the database copies with update DB copy or by removing and re-adding.

Note that in your above cmdlets I did not see the -configurationonly parameter.  That is need if the server is not functioning.  For example see:

https://technet.microsoft.com/en-us/library/dd297956%28v=exchg.150%29.aspx?f=255&MSPPError=-2147217396

Hey Rhoderick,

Thats what i was hoping to be able to do.

Note that in your above cmdlets I did not see the -configurationonly parameter. That is need if the server is not functioning. For example see: https://technet.microsoft.com/en-us/library/dd297956%28v=exchg.150%29.aspx?f=255&MSPPError=-2147217396

So is the -configurationonly parameter needed if the "exchange install" part of the offline server is working?

The array rebuild did not work (i do have 1 drive left with the working array config but am ready to move past this).

So i just want to make sure i am on the same page. The server that went offline has 2 arrays, 1 for c drive and 1 for d drive. Exchange is installed on the c drive. D drive housed the DB only. The d drive is the drive that was lost. So when i go to run /recoverserver, will the installed exchange conflict with /recoverserver?

• Edited by forgiven Thursday, July 09, 2015 12:09 PM

@Ed -- Yarp!!

Mr/Mr/Ms Forgiven, did you read Example 2 and expand the section that explains parameters?

Now that i have removed the DB copies and the server from the DAG, do i add it back to the DAG and then add the DB copy or update the out-of-date copy with the up-to-date copy? This is all based on the VHDX working and being put back in the same location on the drive.

Or maybe it is safer to just reseed?

• Edited by forgiven 7 hours 46 minutes ago

Now that i have removed the DB copies and the server from the DAG, do i add it back to the DAG and then add the DB copy or update the out-of-date copy with the up-to-date copy? This is all based on the VHDX working and being put back in the same location on the drive.

Or maybe it is safer to just reseed?

• Edited by forgiven Thursday, July 09, 2015 11:43 PM

• Edited by forgiven 19 hours 50 minutes ago

• Edited by forgiven Friday, July 10, 2015 11:37 AM

Marking posts as answers and helpful is all I ask.  I'm happy to have helped.