Hi, we have a load balance BizTalk server and clustered SQL in production

Where as in DR there was on application and database server.

DR server is joined to production SSO and Group and implemented a log shipping as DR solution. On DR database we can always see databases in restore state.

We have a requirement from client that production system should not be stopped during DR exercise. 

Now during DR exercise I need to update the database on DR SQL server using ssomange -updatedb command, as the server is joined to production system I can't stop Production SQL as DR has a dependency on Production.

But I am worried about real disaster case, there is no SQL server in real disaster case, so how could I use ssomanage on DR server. 

An error occurred while attempting to access the SSO database.
 Function: GetDeletedMappings
 File: credentialcache.cpp:968
 A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server).
 SQL Error code: 0x00000035
 Error code: 0xC0002A21, An error occurred while attempting to access the SSO database.

I can't use secret file on DR from production, with out issuing ssomanage. 

If I try to restore production master secret file on DR using ssoconfig -restoreSecret keyfile.bak I can see below error.

The secret key can be stored on master secret server only. So how to make my DR as a master secret server when there is no production system available, but the DR is SSO and Group joined to Production. 

Please advise.

First, we have to say that the DR exercise should wait for a planned downtime.  Anyway...

What you're describing should work just fine provided you stop the Backup job on the prod system during the exercise.  However, this is why you wait because technically, you won't have a DR site during the DR exercise.

The process to bring up DR would be exactly the same regardless of whether PROD is still running or not.

You must follow all of the steps exactly in the guides.  For example, the sso .bak file should already have been applied to the DR SSO Servers and a copy kept accessible just in case.

Also, you may have missed the step where all the DR servers, including SSO, are re-pointed to the DR SQL Server(s).  Because of that, ssomanage and ssoconfig would be acting against the now running DR databases.

Hi, Thanks for your response.

I am stopping the backup job on production before brining the DR databses online. SSO bak files are copied from prod to dr.

"The process to bring up DR would be exactly the same regardless of whether PROD is still running or not."

This where I got some questions in mind, when the DR is joined to production how could I promote the master secret from Production site to DR site using ssomanage command in a real disater case where your production SQL is no longer available. 

Once the servers are pointed to the DR SQL Servers the, all the BizTalk and SSO services, including the MSS are pretty much unaware of the other site.

The backup/restore process essentially creates a clone of the PROD Group which can run independently.

If the client is insistent on no outage during DR, you should explore the option of having a cluster spanning DC & DR. That will impose a constraint on the existing DC/DR bandwidth but the service failover would ensure you do not need to do the gymnastics associated with DR recovery.

Also if your SSO master was then clustered across the two sites, failing it over to DR would achieve the DR failover for the SSO service. Similarly, failing the SQL instances to the DR DB node would achieve the failover for the databases. Similarly if the Front-end recieve locations are clustered then failover would failover the receive locations only. All this while causing the normal service disruption during the failover would however not require outage.

Regards.

Hi Shanky,

Thanks for your advise, at this moment cluster spanning between DC and DR is not possible. 

Is there anyway we can achieve below.

When the DR is joined to production how could we promote the master secret from DC to DR using ssomanage command in a real disaster case where your production SQL is no longer available. 

Can anyone please help on this ?

The SSO in BizTalk is a combination of the Enterprise SSO Service and the SSO Database. The Master promotion is doable ONLY after the Databases become available.

In any BizTalk DR Scenario, the first action is recovery of BizTalk Databases (this is done by running the RecoverToMark job created during the BizTalk Log Shipping setup). This is then followed by updating the server names for the BizTalk Front-ends. Then you bring up the Enterprise SSO service and make one of the DR nodes the SSO master using the ssomanage commands. Then the BizTalk Services (host instances) are brought online and then the suspended instances will auto resume.

Regards.

While technically possible, clustering across sites is not a supportable, nor a supported, option.

Sorry if I'm not explaining well enough.

First, I'll point out that the step you're asking about is not part of the DR recovery process at all.  There is no part of recovery that requires the PROD site to be accessible.

During recovery, all the servers, including SSO, are repointed to the DR SQL Servers.  The steps for SSO are not very well documented.  I think the sample scripts are the only place you see this done.

Here's another thread that has the general process.  http://serverfault.com/questions/273125/how-can-one-recover-sso-from-a-disaster

ssoconfig -setdb is what changes where the SSO Service looks for it's database

ssomanage - updatedb is what changes the location of the MSS in the SSODB.

So basically, at recovery, you:

• Repoint all the SSO Services, including the MSS, to the DR SQL Server.
• Restore the Master Secret on the DR MSS node(s).
• Update SSODB to point to the DR MSS

Hi,

The below link might help you with the required steps in BizTalk Disaster Recovery.

http://www.codeproject.com/Articles/529862/BizTalkplusServerplusDisasterplusRecovery

Thanks,

Sumit

Thank you all for your response, let me try with these options first. 

I will come back on this thread shortly. Thanks again.

Hi All,

Just thought of updating the issue, basically I have performed the below steps on DR, with some changes in SSO table.

• Repoint all the SSO Services, including the MSS, to the DR SQL Server.
• Restore the Master Secret on the DR MSS node(s).
• Update SSODB to point to the DR MSS

These are the steps I executed to recover DR when DR BizTalk server joined to production group and without bringing the production server down. 

1. Updated the front ends and database names on DR application server and restarted ENTSSO, WMI service

2. On DR database server issued series of commands 

(Updated the dbo.SSOX_GlobalInfo table entry from production server to DR server)

USE SSODB

UPDATEdbo.SSOX_GlobalInfo SET gi_SecretServer='DRDBSERVER'

Now issued following commands to make the DRDBSERVER as master secret

ssomanage -serverall MASTERSERVER.XML

ssoconfig -setdb DRDBSERVER

ssoconfig -restoresecret BTSSO.BAK

3. Restarted ENTSSO and WMI service on DRDATABASE server.

where MASTERSERVER.XML contains below

sso>
  <globalInfo>
     <secretServer>DRDBSERVER</secretServer>
  </globalInfo>
</sso>

And BTSSO.BAK is a secret file copied from production. 

Now, launched BizTalk admin console and refreshed it couple of times, I don't find any issues and master secret server name updated to DRDBSERVER on SSO GUI.

Please note that all these steps executed on DR site with out bringing the production SQL down, also checked production site, I din't see any issues. All is ok.

During the real disaster case, I still follow the same recovery procedure but with below sso commands. (With out doing any changes on SSO tables) 

ssomanage -updatedb MASTERSERVER.XML

ssoconfig -restoresecret BTSSO.BAK

Glad it worked out.  However....:)

You should not have to update the SSODB directly as noted in step 2.*

ssomanage + ssoconfig should give you the same result.

If you have the oppty to try this again and get an error with SSO, please post that and we can help.

*Depending on how your host names are configured in DNS, you may have to also update the SSO Server name in the BizTalk Group Properties.  That may be what you updated in the db, sorry, I don't recall exactly.  If that's the case, you should make that change in BizTalk Administrator, though it is effectively the same thing.

Initially I tried with ssomanage and ssoconfig, but some how sso is not updated in tables, I have double checked, not sure I face this some issue.

When I issue the below commands with out updating the tables there is no issues.

ssomanage -serverall MASTERSERVER.XML

ssoconfig -setdb DRDBSERVER

But when I run the ssoconfig -restoresecret command, I received "secrets can only be stored on master secret server"

As there is no option I must go to database and updated the value manually. Later when I issue ssoconfig -restoresecret command it's working 

ssomanage -serverall MASTERSERVER.XML

This is not part of the BizTalk Recovery process.

You should be doing ssoconfig -setdb first to change the pointer to SSODB, then ssomanage -updatedb to change the MSS.Other than manually stopping the Backup job on the Production kit, there should be no difference in your process to bring DR online whether Prod is available or now.