We have a customer who has 2 AD sites, 1 in San Francisco and 1 in Hong Kong.

Both sites have Exchange 2013 Std CU2 installed (Win2012 Std), single server running both the CAS and MBX roles.

A single DAG exists that spans both sites. 

Each server has 2 Active Databases and 2 Passive Databases in each site.

The witness server is in a 3rd site.

There is an MPLS connection between these 2 sites and they both have connectivity to the internet.

There are 2 Send Connectors: 

1.  SF Outbound to Internet, Address Space *, Cost 3, Source server:  SFServer (only)

2.  HK Outbound to Internet, Address Space *, Cost 4, Source server:  HKServer (only)

The customer would like the SF Outbound to Internet Send Connector to be the primary path for outbound mail - as it has a lower cost.  If the SF server has any type of outage, than the HK send connector should take over sending outbound mail - automatically.

Currently, it appears that when a user with a mailbox in the Hong Kong site sends a message, it goes out the internet directly from the Hong Kong server, using the HK Outbound to Internet Send Connector, instead of using the SF Outbound to Internet connector, which has only the SF Exchange server as a source server. 

I'm confused why this is happening because the SF Outbound to Internet has a lower cost, so shouldn't all mail go out this connector by default?

And if this is not the designed behavior, what are the options to provide for the desired configuration, which is...all mail goes outbound via the SF server and only goes outbound via Hong Kong if SF server is unavailable?

Any help would be greatly appreciated.

Your HK traffic is likely routing out HK instead of SF because that is equivalent to the least cost route. From HK to the Internet is 4, from HK to SF is at least 1, and from SF to the Internet is 3. Therefore, for HK to get to the internet through SF, it costs 1 to get to SF, and then 3 to get to the Internet for a total of 4. If these are the only 2 exchange sites, try setting the costs to have a greater difference. In a *over scenario, in order to shape traffic the way that you would like, you will have to modify the costs in order to allow mail to flow the least cost route. Scripts, or a product like SCOM can do this for you "automatically" as to avoid manual intervention.

Ok yes, that makes sense.  So I reviewed the Site Links and costs and here's what I found. 

There are actually 4 AD Sites, but only 2 with Exchange servers. 

They are all connected with each other with the Default IP Site Link which has a cost of 300 (not sure why it was changed to that). 

Considering the Default IP Site Link cost of 300 between the HK and SF sites and because the Send Connector cost can only be a value between 1-100, modifying the Send Connector links is not going to work.

Looking up the Set-ADSiteLink cmdlet, it says setting the -ExchangeCost parameter configures Exchange to use this "exchangecost" value instead of the AD assigned. 

If that's the case then if I set the Default IP Site Link -exchangecost 25, that should essentially make all the link costs 25 (because they all use the one Default IP Site Link). 

If I set the SF Send Connector to 5 and set the HK Send Connector to 75, then the least cost path would be the Default IP Site Link (modified with the Exchange cost) of 25 + SF Send Connector of 5 = 30, which would be less than the HK Send Connector of 75, so mail should flow out the SF connector, unless it is unavailable and then it would go out the HK connector.

Does that seem logical?  Am I missing anything?  Just hoping for a sanity check

Your math appears to be correct, however costing in Exchange routing is not a high availability feature. Exchange will always want to route via the least cost route. In order to shape the traffic to go out HK instead of SF in the event of a failure, you'll have to update the costs to make HK the least cost route. A script that runs as a scheduled task every so often that detects a failure and makes cost adjustments accordingly can be a viable option to accomplish this as an "automated" solution. Also, you can set up rules & custom code in SCOM to change the costs, if SCOM exists in your enviornment. Refer to the following TechNet article for further reading:

Mail Routing

The article specifically calls out Exchange 2013, but contains information regarding 2010 and 2007 a

I have a few more questions based on the behavior we are observing.

In the environment described above, there is still only the DefaultIPSiteLink with a Cost of 300 and all 4 sites use that link.

There are 2 Send Connectors however, we've added both servers as Source Servers as follows:

SF Outbound to Internet, *, Cost 3, Source Servers SFExch01, HKExch01

HK Outbound to Internet, *, Cost 4, Source Servers SFExch01, HKExch01

I have just tested sending email with internal and external clients  - with mailboxes hosted in databases in each site.  Mail from SF mailboxes goes outbound via the SF server as expected, however messages from HK mailboxes are going from the HK server to SF server and then outbound. 

Based on the least cost routing rule, this should not be happening.  In the very beginning of this post, it was not happening, messages would go out SF and HK respectively based on the location of the mailbox. 

The difference is that both servers are added as Source Servers in each connector.  There is a warning thrown when adding the server from another site, but it is still allowed. 

I expected to see the messages load balanced across the 2 servers, so that mail would be flowing out both locations, but so far, it seems to be picking the SF server.  I have closed the clients, flushed DNS Cache in attempt to force it, but it still continues to send mail from HK to SF and then out.

I'm baffled by the way it's ignoring the Costs.  From HK out is 4, from HK to SF and out is 300 + 3=303

Would this configuration provide HA?  Or is it technically wrong to have both source servers added? 

Additional Information:

Installed CU3 on both servers on Sat 3/1.  Ran all the appropriate maintenance mode commands before and after the install - successfully on both servers.

All services were running on both servers, all databases were healthy, could not find any apparent errors.

However, this morning after reviewing the Application logs, found events indicating that the Front End Hub Transport and the Hub Transport component states were "inactive" on the Hong Kong server.  Running the Get-servercomponentstate showed all the components as "Active."  Restarted the FE Hub Transport and the Hub Transport services.

So the post from above was incorrect.  Mail was flowing out the San Francisco server only because there was an issue with the transport service on the Hong Kong Server.  Mail is currently now flowing as it was previously.  If the message originates in Hong Kong, it goes out that connector, same for San Francisco...least cost path...

I'm still wondering if it's technically wrong to have both servers as source servers in each connector and if this will cause any issues?