I am facing RPC Server Unavailable issues in Win2k3 R2 sp2 Machines where Citrix Xenapp is installed. We had change in IP's of DC ,DNS and Member Servers in our Environment recently. Prior to that everything worked fine. Brief Environment Architecture : We have a firewall . Citrix , db serves and some application boxes are inside the firewall and some app boxes are out the firewall. Policy is we can open the shares from our local machines to out the firewall boxes and from outside firewall boxes we can open the shares of inside firewall boxes. After IP change : We have created Forward and Reverse Lookup Zones for servers. The servers which getting RPC server unavailable error is on servers which are inside the firewall. (There is no change in the environment only IP's range has been changed) The Servers are citrix boxes and they are accepting RDP connections( 3-4 logins irreverently ) and later suddenly unable to RDP the boxes. Error : The system cannot log you to the following error . THE RPC SERVER IS UNAVAILABLE. This is what happening in all the citrix boxes in our test environment ( 3 boxes ) . After rebooting again they are accepting the connections. Below are the steps we have performed after IP change : We have registered DNS. ( ipconfig /registerdns)Synced the time. (w32tm /resync )Checked all services.Disjoined and rejoined the servers to domain. Below are the error messages in logs on citrix servers we are connecting : Systen Logs : EVENT ID : 5719 - This computer was not able to set up a secure session with a domain controller in domain ****** due to the following: The RPC server is unavailable. This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator EVENT ID : 10009 - DCOM was unable to communicate with the computer 10. **. ** .** using any of the configured protocols. EVENT ID : 10016 - The application-specific permission settings do not grant Remote Activation permission for the COM Server application with CLSID {EC141AA5-2FCB-4A70-BBA0-025B6A2A626E} to the user NT AUTHORITY\ANONYMOUS LOGON SID (S-1-5-7). This security permission can be modified using the Component Services administrative tool. EVENT ID :10006 - DCOM got error "General access denied error " from the computer when attempting to activate the server: {EC141AA5-2FCB-4A70-BBA0-025B6A2A626E} Application Logs : EVENT ID : 1219 - Logon rejected for Domain\user1. Unable to obtain Terminal Server User Configuration. Error: The RPC server is unavailable. EVENT ID : 1053 - Windows cannot query for the list of Group Policy objects. Check the event log for possible messages previously logged by the policy engine that describes the reason for this. EVENT ID : 1000 - Could not execute the following script \\domain\SysVol\taxstream.local\scripts\logoff.bat. No network provider accepted the given network path. Guys, Please guide us on the above issues. Thanks in Advance. Chaitanya Varma MCTS

Hello, first assure that no firewalls are blocking required ports for AD and connectivity http://technet.microsoft.com/en-us/library/dd772723(WS.10).aspx Additional it is important that only the domain DNS servers are used on machines NICs and DCs are NOT multihomed, more then one ip address or NIC is used. Did you check the DNS zones and SRV records that all DCs are registered with the correct ip addresses and did you also update AD sites and services for the new subnets? For broken security channels often imaged machines are a reason that are not prepared with sysprep, so how machines are installed in your domain?Best regards Meinolf Weber MVP, MCP, MCTS Microsoft MVP - Directory Services My Blog: http://msmvps.com/blogs/mweber/ Disclaimer: This posting is provided AS IS with no warranties or guarantees and confers no rights.

Thanks for your reply Weber. 1) Do I need to check all the ports mentioned in the article ,whether I can able to telnet them from my local system? 2) We have done NIC teaming and (DNS is Installed in our DC only ) so provided Preferred DNS as DC 's IP and alternate DNS as ( public DNS for internet ) ? 3)we have just deleted the previous zones and recreated them ( forward and reverse ) apart from this do I need to check anything ? FYI : servers are accepting connections and suddenly we are getting RPC error , after some time automatically again without rebooting we can relogin . now also same thing happened and below are the logs : EVENT ID : 1219 : Logon rejected for Domain\chaitanya. Unable to obtain Terminal Server User Configuration. Error: The RPC server is unavailable. EVENT ID : 5719 - This computer was not able to set up a secure session with a domain controller in domain due to the following: The RPC server is unavailable. This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator. Please put some light on this ? Please could you guide me what all I need to check ? Do I need to contact firewall team on this ? Thanks , Chaitanya. Chaitanya Varma MCTS

Hello, 1. that article contains the MUST ports that are required for AD and yes they MUST be open, telnet allone is too less. 2. NIC teaming on DCs is ONLY supported if configured for failover NOT load balance. DO NOT use the ISPs DNS server on any domain machines, this result in multiple domain problems with connectivity. ISPs DNS server should ONLY be configured as FORWARDERS in the DNS management console in the DNS server properties. So please remove the ISPs DNS server and run ipconfig /flushdns and ipconfig /registerdns and restart the netlogon service on DCs or reboot domain machines machines.Best regards Meinolf Weber MVP, MCP, MCTS Microsoft MVP - Directory Services My Blog: http://msmvps.com/blogs/mweber/ Disclaimer: This posting is provided AS IS with no warranties or guarantees and confers no rights.

we are unable to telnet the ports from our local machines ? we have used nic teaming in our servers for failover. we have not used ISP's dns. Chaitanya Varma MCTS

Hi Chaitanya, We are experiencing the exact same issues as you have described. Same event id's in both System log and Application log. We have a Windows Server 2003 R2 x64 Enterprise environment, running XenApp 5. It all started happening about 2 weeks ago. One server a day typically (a different one each day) would be loaded with between 50 and 70 user desktop sessions, running fine, and all of a sudden those users cant access network resources any more. All sessions still remain connected, but new sessions cannot be established. Users trying to login receive message "Domain cannot be contacted", and Citrix clients being directed to that server receive "Protocol driver error". The first event in the application event log is typically 5719, and then the other events come after that. Its as if some buffer is being filled, but there is nothing specific in the event log to suggest what it is. A few hours later, it comes good once again. No reboot necessary. It is not a load issue, these servers are Dell M710 Blades, with dual proc, 6 core Intel x5675 with 96gb Ram and SSD's in Raid 1. Nics are Broadcom 5709S. We have removed the Teaming and updated to the latest drivers, but the issue still exists. I have tried disabling TCP Offload, TCP Chimney in the OS, as well as RSS on the NIC. None of these changes seem to make a difference. This is causing us major grief as users that are on that particular server have to log off and back on to another server to continue working. When they log off, their roaming profiles are not updated (because the computer cant communicate with the domain controller). Any help would be much appreciated! Daniel

Thanks Yagmoth for that information on how RPC works, its very interesting. Our servers are all inside the corporate network, and Windows Firewall is off on all the servers, so i dont think its Firewall in our case. When i perform a netstat - ano on the server, i get alot of ports in use right up to 6000 or so, but that means theres still alot free if 65534 is the limit? Could there be some other TCP related buffer which is getting filled up other then RPC ports?

Ok, i appear to have had some success with this within our environment, and thought i'd should share my solution with you all. Thanks to everyone who had replied, and particularly Yagmoth who inspired the idea that RPC ports could be responsible. We added the following registry keys: KEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters "TcpTimedWaitDelay": dword: 0000001e "MaxUserPort": dword: 0000fc00 Since applying these keys, and rebooting the servers, we have not seen a repeat of the event and it has been a week now. We were having this every day at least once a day, so it looks promising. These articles are good reference: http://en.competence.winterheller.prevero.com/index.php?title=FAQ:What_do_the_three_registry_keys_TCPTimedWaitDelay%2C_MaxUserPort_and_MSSQLStatistics_mean%3F#tab=Until_Windows_XP http://social.technet.microsoft.com/Forums/en/winservergen/thread/8ac3b706-17de-480c-bf8e-1a719595cc7a To summarise, by default Windows allows 5000 simultaneous open TCP/UDP ports, and will hold them open when inactive for a period of 2 minutes before it closes them. Something or someone (being a Citrix environment, it could be something a user is doing) was causing those 5000 ports to be all open, and other system services were failing due to this. Although it was probably only for a brief moment, this is enough for these other services on the system to fail and not recover for several hours. These changes increase the amount of ports Windows can use to 20,000 and reduced the time Windows holds those ports open to 30 seconds. Theoretically we could have gone right up to 50000 ports, but unless there is a reason for this, i thought that is a bit excessive. See how it goes i guess. Netstat -ano was very helpful in identifying these open ports.