Citrix XenApp: Sessions disconnected right after connection

Written by Ingmar Verheij on August 7th, 2013. Posted in Remote Desktop / Terminal Server, XenApp (Presentation Server)

When new sessions are started, either via Microsoft RDP of Citrix ICA, they are disconnected within seconds. This applies to normal users and users with administrative privileges. This problem is caused by a chain of events. One components crash leads to an ungraceful shutdown of other components leaving a garbage configuration, preventing new connections.

Symptoms

Users connect to a Citrix XenApp server and are immediately disconnected. No sessions are visible on the server (no disconnected sessions either) and the same symptoms apply for sessions via Microsoft RDP. There are no events logged in the event log that indicate a potential problem.In some occasions the first RDP/ICA session is successful and subsequent sessions fail.

 

Situation

Consider the following scenario:

  • Remote Desktop Session Host role (RDSH) is installed
  • Require secure RPC communication is enabled for RDSH
  • Set client connection encryption level is configured for RDSH
  • There are active (or disconnected) sessions either via RDP or ICA

 

Chain of events

The following events occur causing a chain of events leading to the described symptoms.

Group policy refresh

Every 90 minutes group policies are automatically refreshed (or manually via gpupdate /force). When this happens the Remote Desktop service reloads to load the GPO changes.

In the Application log we can see an event is raised by SceCli (Security Configuration Editor Client for Windows) with ID 1704 informing us that a new security policy is applied successfully.

Event 1704, SceCli

 

Remote Desktop services crash

When the Remote Desktop service restarts it unloads and reloads the module winsta.dll (the module that handles the WinStations), but reloads it incorrectly. As a result the Remote Desktop services will crash.

First an event is raised by Application Error with ID 1000 informing us that the application svchost.exe_TermService (the process of the Remote Desktop service) has an error. The faulting module is WINSTA.dll_unloaded.

Event 1000, Application Error

90 seconds after the Remote Desktop Service stops working the winlogon notification subscriber notices the <TermSrv> is taking a long to handle notification events, as can be seen in an event from Winlogon with ID 6005.

Event 6005, Winlogon

Around 5 minutes later the Citrix Health monitors notice something’s wrong, both the Terminal Services test and the Ticketing test fail. Two error events are raised by CitrixHealthMon with ID 2005.

Event 2005, CitrixHealthMonEvent 2005, CitrixHealthMon

Since the Remote Desktop Service is no longer working the Citrix Health monitors can’t perform a recovery action (disabling logon). A warning is logged from CitrixHealthMon with ID 1001.

Event 1001, CitrixHealthMon

After 1 hour and 1 minute <TermSrv>  handles the notification event sent by the winlogon notification subscriber.

Event 6006, Winlogon

Sure enough, not much later the Citrix Health monitor was able to successfully run and pass the Terminal Services test and the state and failure threshold has been reset. Am information event is logged from CitrixHealthMon with ID 2006.

Event 2006, CitrixHealthMon

 

Remote Desktop services starts

The Service Control Manager detects when services fail and perform recovery actions. All services are configured by default to restart the service after 1 minute.

Remote Desktop Services - Recovery

Although the service seemed to start working correctly again (the notification events are received and the health checks are passed) the Service Control Manager detects that the Remote Desktop Service is terminated unexpectedly and it will perform a correction action (Restart the service) in 60000 milliseconds, or 1 minute. Notice this happens 1 hour after the initial problem started.

Event 7031, Service Control Manager

Sure enough an event is raised by Service Control Manager with ID 7036 informing us that the Remote Desktop Services service has entered the running state.

Event 7036, Service Control Manager

At the same time an event is raised by DistributedCOM with ID 10010 informing us that the server {F9A874B6-F8A8-4D73-B5A8-AB610816828B} did not register in time. The server referred to is the “Terminal Services Connection Manager Class”, which makes sense if the Remote Desktop Services service is restarted.

Event 10010, DistributedCOM{F9A874B6-F8A8-4D73-B5A8-AB610816828B}

 

 

 

 

Sessions (RDP/ICA) are reset

Since the Remote Desktop Service is restarted the active (and disconnected) sessions are reset. In a normal situation an event is sent to all depending services such as Citrix IMA (used in Citrix XenApp 6.5) to inform when sessions are ended. When Citrix receives receives an event it can do some housekeeping such as cleaning up the Sessions key in the registry.
HKLM\SOFTWARE\Citrix\Ica\Session

Since this event is never sent (the Remote Desktop Services service is in an inconsistent state and restarting) no housekeeping takes places and garbage remains in the registry. In this example three sessions where active. ID 3, 4 and 14.

Active sessions

 

Citrix XenApp reloads

Citrix XenApp detects that Remote Desktop Services is restarted and will act accordingly. For instance it will contact the Citrix license server to verify if it is available. An informational event is raised by MetaFrame with ID 9019.

Event 9019, MetaFrame

What Citrix will not do is housekeeping, not even when a server restarts. As a result the garbage of the previous sessions (ID 3,4 and 14) remain in the registry causing problems for new sessions.

 

New sessions

So what happens if a new session is initiated? Since Citrix has hooked into Remote Desktop Service it is informed immediately when new Winstations are created (which happens for both RDP and ICA). Citrix is kind enough to keep track of all sessions in the HKLM\SOFTWARE\Citrix\Ica\Sessions key, regardless of the protocol used. In other words, RDP sessions are stored in the Ica sessions key as well.

As a result the following scenario could occur (example):

  • A  user initiates a new session via Microsoft RDP
  • A new Winstation is created with ID 3
  • Citrix hooks into the new WinStation creation and tries to register the session
  • It detects that ID 3 already exists and this is an ICA session
  • Citrix and Remote Desktop Services are confused, what happened? Must be a glitch in the network or somehome hacking into the system. Better safe then sorry ,  let’s end this session right now!

 

Solutions

For this problem there are two solutions. One is already in place, the other needs to be implemented.

  • The described problem with the Remote Desktop Services host crashing after refreshing group policies (with increased security) is a know bug that’s fixed with a hotfix. It is described in KB2479710.
  • Citrix needs to add a housekeeping task in Citrix (XenApp) and clean all known sessions on a restart. This needs to be implemented in a future version (or hotfix).

Both Microsoft and Citrix need to increase the events generated in the scenario described above. You can’t disconnect a session without raising an event in the event log describing why.

 

 

.

Ingmar Verheij

At the time Ingmar wrote this article he worked for PepperByte as a Senior Consultant (up to May 2014). His work consisted of designing, migrating and troubleshooting Microsoft and Citrix infrastructures. He was working with technologies like Microsoft RDS, user environment management and (performance) monitoring. Ingmar is User Group leader of the Dutch Citrix User Group (DuCUG). RES Software named Ingmar RES Software Valued Professional in 2014.

More Posts - Website

Follow Me:
TwitterLinkedInGoogle Plus

Tags: , , ,

Comments (3)

  • carl webster
    7 August 2013 at 13:14 |

    Group Policy is refreshed every 90 minutes plus a random offset of up to +30 minutes. So actually every 90 to 120 minutes.

    http://technet.microsoft.com/en-us/library/cc940895.aspx

    Webster

  • Veronica
    8 August 2013 at 09:47 |

    Hey Ingmar,
    Thanks a lot for such an informative and helpful article.I’ve faced the same problem of being disconnected by the Citrix XenApp server in a very short time after connecting to it.Since then I was looking for its issue.Thanks again.

  • Patrick
    16 June 2014 at 15:26 |

    Hi Ingmar,

    I hope you could help me with a XenDesktop/XenApp (7.5) issue. I tried to publish a W7 Machine and a Windows Server Machine over Citrix Storefront on my Linux Thin Client. I put in the Connection Details for a XenServer, I activated DNS, I activated ICA Receiver 13. So far so good, I was able to publish the desktops I mentioned above. But when I try to start the Desktops I always got a error message: No Server Connection configured
    Do you have a Solution to this issue? I also got the error message: The section “Application Server” is missing in the Configuration file.

    Thanks for your Effort!
    Patrick

Leave a comment

*

Donate

%d bloggers like this: