Incoming mail flow problems on CU6 multirole server

Hello,

I have a newly-installed Exchange 2013 SP1 server (all roles installed on one server) that I installed CU6 on, and now the incoming mail flow has stopped.  Incoming mail is rejected at SMTP time with the following message:

"451 4.7.0 Temporary server error. Please try again later. PRX5"

From the logs, I traced it back to the Transport Service "Default SERVERNAME" receive connector, which is configured to listen on 0.0.0.0:2525.  Here are the relevant log snippets:

D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive\RECV20150321-1.LOG:
2015-03-21T17:13:07.697Z,excambio\Default Frontend EXCAMBIO,08D2320EEE2DB8E8,30,127.0.0.1:25,127.0.0.1:23768,*,,Proxy destination(s) obtained from OnProxyInboundMessage event
2015-03-21T17:13:07.697Z,excambio\Default Frontend EXCAMBIO,08D2320EEE2DB8E8,31,127.0.0.1:25,127.0.0.1:23768,*,,NextHopFqdn property is null or whitespace when creating InboundProxyLayer
2015-03-21T17:13:08.743Z,excambio\Default Frontend EXCAMBIO,08D2320EEE2DB8E8,32,127.0.0.1:25,127.0.0.1:23768,*,,"Message or connection acked with status Retry and response 441 4.4.1 Error encountered while communicating with primary target IP address: ""Failed to connect. Winsock error code: 10061, Win32 error code: 10061."" Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts. The last endpoint attempted was 192.168.27.25:2525"
2015-03-21T17:13:08.743Z,excambio\Default Frontend EXCAMBIO,08D2320EEE2DB8E8,33,127.0.0.1:25,127.0.0.1:23768,>,451 4.7.0 Temporary server error. Please try again later. PRX5 ,

D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpSend\SEND20150321-1.LOG:
2015-03-21T17:08:07.749Z,Inbound Proxy Internal Send Connector,08D2320EEE2DB8E2,0,,192.168.27.25:2525,*,,attempting to connect
2015-03-21T17:08:08.778Z,Inbound Proxy Internal Send Connector,08D2320EEE2DB8E2,1,,192.168.27.25:2525,*,,"Failed to connect. Winsock error code: 10061, Win32 error code: 10061, Error Message: No connection could be made because the target machine actively refused it 192.168.27.25:2525"
2015-03-21T17:08:12.397Z,Client Proxy Send Connector,08D2320EEE2DB8E4,0,,192.168.27.25:465,*,,attempting to connect. Client proxy session for HealthMailbox9301776481cb41068c2afd4f62f32744@jonheese.com

D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\Connectivity\CONNECTLOG20150321-1.LOG:
2015-03-21T17:13:07.697Z,08D2320EEE2DB8E9,SMTP,internalproxy,+,Undefined 00000000-0000-0000-0000-000000000000;QueueLength=<no priority counts>
2015-03-21T17:13:07.697Z,08D2320EEE2DB8E9,SMTP,internalproxy,>,excambio.jonheese.local[192.168.27.25]
2015-03-21T17:13:08.743Z,08D2320EEE2DB8E9,SMTP,internalproxy,>,Failed connection to 192.168.27.25:2525 (ConnectionRefused:0000274D)[TargetHost:excambio.jonheese.local:2525|MarkedUnhealthy|FailureCount:5|NextRetryTime:2015-03-21T17:13:08.778Z][TargetIPAddress:192.168.27.25:2525|MarkedUnhealthy|FailureCount:5|NextRetryTime:2015-03-21T17:13:08.778Z]
2015-03-21T17:13:08.743Z,08D2320EEE2DB8E9,SMTP,internalproxy,-,Messages: 0 Bytes: 0 (Retry : Unable to connect)

When I try telnet'ing to localhost:2525 (or the actual IP of the exchange server), I either get an initial connection that drops after I hit any key, or I get connection refused.

I have confirmed that the "Default SERVERNAME" connector is configured appropriately, under the HubTransport role, listening on 0.0.0.0:2525:

[PS] C:\Windows\system32>Get-ReceiveConnector "excambio\Default EXCAMBIO" | fl


RunspaceId                              : 73ca4d69-3185-41ac-abff-a82be444730c
AuthMechanism                           : Tls, Integrated, BasicAuth, BasicAuthRequireTLS, ExchangeServer
Banner                                  :
BinaryMimeEnabled                       : True
Bindings                                : {[::]:2525, 0.0.0.0:2525}
ChunkingEnabled                         : True
DefaultDomain                           :
DeliveryStatusNotificationEnabled       : True
EightBitMimeEnabled                     : True
SmtpUtf8Enabled                         : False
BareLinefeedRejectionEnabled            : False
DomainSecureEnabled                     : False
EnhancedStatusCodesEnabled              : True
LongAddressesEnabled                    : False
OrarEnabled                             : False
SuppressXAnonymousTls                   : False
ProxyEnabled                            : False
AdvertiseClientSettings                 : False
Fqdn                                    : excambio.jonheese.local
ServiceDiscoveryFqdn                    :
TlsCertificateName                      :
Comment                                 :
Enabled                                 : True
ConnectionTimeout                       : 00:10:00
ConnectionInactivityTimeout             : 00:05:00
MessageRateLimit                        : Unlimited
MessageRateSource                       : IPAddress
MaxInboundConnection                    : 5000
MaxInboundConnectionPerSource           : 20
MaxInboundConnectionPercentagePerSource : 2
MaxHeaderSize                           : 128 KB (131,072 bytes)
MaxHopCount                             : 60
MaxLocalHopCount                        : 12
MaxLogonFailures                        : 3
MaxMessageSize                          : 35 MB (36,700,160 bytes)
MaxProtocolErrors                       : 5
MaxRecipientsPerMessage                 : 200
PermissionGroups                        : ExchangeUsers, ExchangeServers, ExchangeLegacyServers
PipeliningEnabled                       : True
ProtocolLoggingLevel                    : Verbose
RemoteIPRanges                          : {::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, 0.0.0.0-255.255.255.255}
RequireEHLODomain                       : False
RequireTLS                              : False
EnableAuthGSSAPI                        : False
ExtendedProtectionPolicy                : None
LiveCredentialEnabled                   : False
TlsDomainCapabilities                   : {}
Server                                  : excambio
TransportRole                           : HubTransport
SizeEnabled                             : Enabled
TarpitInterval                          : 00:00:05
MaxAcknowledgementDelay                 : 00:00:30
AdminDisplayName                        :
ExchangeVersion                         : 0.1 (8.0.535.0)
Name                                    : Default EXCAMBIO
DistinguishedName                       : CN=Default EXCAMBIO,CN=SMTP Receive
                                          Connectors,CN=Protocols,CN=excambio,CN=Servers,CN=Exchange Administrative
                                          Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=JonHeese,CN=Microsoft
                                          Exchange,CN=Services,CN=Configuration,DC=jonheese,DC=local
Identity                                : excambio\Default EXCAMBIO
Guid                                    : 018e86b8-351e-4ba5-92f9-f527f8200c36
ObjectCategory                          : jonheese.local/Configuration/Schema/ms-Exch-Smtp-Receive-Connector
ObjectClass                             : {top, msExchSmtpReceiveConnector}
WhenChanged                             : 3/21/2015 12:43:50 PM
WhenCreated                             : 3/21/2015 11:57:44 AM
WhenChangedUTC                          : 3/21/2015 4:43:50 PM
WhenCreatedUTC                          : 3/21/2015 3:57:44 PM
OrganizationId                          :
OriginatingServer                       : artemis.jonheese.local
IsValid                                 : True
ObjectState                             : Unchanged

And I've already tried the hosts file trick, with no change to this behavior.

It's probably worth noting too that this is *not* intermittent behavior -- this happens with every single email that is sent to the frontend (TCP/25) receive connector.

I've also confirmed that my components are all active:

[PS] C:\Windows\system32>Get-ServerComponentState excambio

Server                                  Component                               State
------                                  ---------                               -----
excambio.jonheese.local                 ServerWideOffline                       Active
excambio.jonheese.local                 HubTransport                            Active
excambio.jonheese.local                 FrontendTransport                       Active
excambio.jonheese.local                 Monitoring                              Active
excambio.jonheese.local                 RecoveryActionsEnabled                  Active
excambio.jonheese.local                 AutoDiscoverProxy                       Active
excambio.jonheese.local                 ActiveSyncProxy                         Active
excambio.jonheese.local                 EcpProxy                                Active
excambio.jonheese.local                 EwsProxy                                Active
excambio.jonheese.local                 ImapProxy                               Active
excambio.jonheese.local                 OabProxy                                Active
excambio.jonheese.local                 OwaProxy                                Active
excambio.jonheese.local                 PopProxy                                Active
excambio.jonheese.local                 PushNotificationsProxy                  Active
excambio.jonheese.local                 RpsProxy                                Active
excambio.jonheese.local                 RwsProxy                                Active
excambio.jonheese.local                 RpcProxy                                Active
excambio.jonheese.local                 UMCallRouter                            Active
excambio.jonheese.local                 XropProxy                               Active
excambio.jonheese.local                 HttpProxyAvailabilityGroup              Active
excambio.jonheese.local                 ForwardSyncDaemon                       Active
excambio.jonheese.local                 ProvisioningRps                         Active
excambio.jonheese.local                 MapiProxy                               Active
excambio.jonheese.local                 EdgeTransport                           Active
excambio.jonheese.local                 HighAvailability                        Active
excambio.jonheese.local                 SharedCache                             Active

Any assistance at this point is greatly appreciated.  I've spent 3-4 days just getting this system up and importing 37GB of mailboxes to it -- reinstalling at this point is not something I'm looking forward to doing (not to mention the fact that all incoming email is just queuing up at a backup MX while I'm waiting for this to go live...).  Thanks!

Regards,

Jon Heese


March 21st, 2015 5:23pm

Hi,

From your description, I recommend you add your Exchange server's IP address to your host file, for example:
192.168.16.1 server
192.168.16.1 server.domain.local

And then restart the Microsoft Exchange Frontend Transport service and Microsoft Exchange Transport service to check the result.

For your reference:

https://social.technet.microsoft.com/Forums/exchange/en-US/fc26dac5-d4e2-49da-903d-361ea8b85388/451-470-temporary-server-error-please-try-again-later-prx5?forum=exchangesvrgeneral

Hope this can be helpful to you.

Best regards,

Free Windows Admin Tool Kit Click here and download it now
March 23rd, 2015 5:03am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics