Improper server boot order in a MS Lync/prairieFyre environment causes communication and endpoint creation issues

PROBLEM

MS Lync customers running prairieFyre 5.10 are unable to route call to queues and agents unable to login after they have performed maintenance on their Domain Controller/Active Directory server. 

SYMPTOMS

In YourSite Explorer, under the Queues section, you will notice that all queues are grayed out.  This typically means they are not provisioned in Lync.  Also, Ignite users will be unable to login successfully.  They will either see the Ignite bar just showing “Loading…”, or if they get logged in, there will just show question marks for their timer (???).

Below is a snippet of the logging from the Provisioner Log file, which indicates that communication to the Active Directory Server (DC) is unavailable:

eVerbose  7/23/2012 5:14:42 AM     TrustedApplicationPool Successfully executed Get-CsTrustedApplicationPool -Identity pF01.pfyre.com. Returned 0 results.
eWarning  7/23/2012 5:14:42 AM     Provisioner  TrustedApplicationPool pF01.pfyre.com to host prairieFyre.UCMAConnector does not exist. Trying to create it first...
eWarning  7/23/2012 5:14:42 AM     TrustedApplicationPool PowerShell object error: [Microsoft.Rtc.Management.ADConnect.ADTransientException: Active Directory server "PFDC001.pfyre.com" is not available. Try again later.
   at Microsoft.Rtc.Management.ADConnect.Connection.ADConnectionPoolManager.GetConnectionFromPool(ConnectionPoolType connectionPoolType, ADObjectId domain, String serverName, Int32 port, Boolean& serverConnectionPresentButDownOrDisconnected)
   at Microsoft.Rtc.Management.ADConnect.Connection.ADConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential)
   at Microsoft.Rtc.Management.ADConnect.Connection.ADConnectionPoolManager.GetConnection(ConnectionType connectionType, NetworkCredential networkCredential, String serverName, Int32 port)
   at Microsoft.Rtc.Management.ADConnect.Session.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, ADObjectId& rootId)
   at Microsoft.Rtc.Management.ADConnect.Session.ADSession.Find(ADObjectId rootId, String optionalBaseDN, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator, Boolean includeDeletedObjects)
   at Microsoft.Rtc.Management.ADConnect.Session.ADSession.Find(ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCtor, CreateObjectsDelegate arrayCtor)
   at Microsoft.Rtc.Management.ADConnect.Session.ADSession.Find[TResult](ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties)
   at Microsoft.Rtc.Management.ADConnect.Session.ADSession.GetTopologySetting()
   at Microsoft.Rtc.Management.ADConnect.Session.ADSession.GetBackEndServer()
   at Microsoft.Rtc.Management.Xds.ManagementConnection.GetConnectionFromActiveDirectory(OcsCmdlet cmdlet, ADSession session)
   at Microsoft.Rtc.Management.Xds.ManagementConnection.<>c__DisplayClass2.<SetupConnection>b__0()
   at Microsoft.Rtc.Management.Internal.Utilities.DeImpersonator.Run[T](Boolean dropImpersonation, Func`1 func)
   at Microsoft.Rtc.Management.Xds.ManagementConnection.SetupConnection(Boolean isLocalStore, OcsCmdlet cmdlet, Boolean shouldDropImpersonation, ManagementConnection& connection, Boolean& ownConnection)
   at Microsoft.Rtc.Management.Xds.XdsCmdlet.CmdletBeginProcessing()
   at Microsoft.Rtc.Management.OcsCmdlet.BeginProcessing()
   at System.Management.Automation.Cmdlet.DoBeginProcessing()
   at System.Management.Automation.CommandProcessorBase.DoBegin()]

CAUSE

The maintenance performed on the Domain Controller/Active Directory server required a reboot, which would have caused a communication failure to the Lync server, as well as the prairieFyre server.  When the Domain Controller/Active Directory server is rebooted, all associated servers will also need to be rebooted afterwards to ensure they reconnect successfully to the DC/AD services. 

RESOLUTION

In the event that the DC/AD server is rebooted (due to maintenance or failure), it is critical to ensure the Lync server is completely rebooted (after the all DC services have come back up).  Then once all Lync Front End services have come online (including pF services), the prairieFyre server should be rebooted.

If this is not possible, restarting the ‘prairieFyre Provisioner Service’ on the Lync server and then the 'prairieFyre Enterprise Router Service' on the prairieFyre server will correct issues with the queues and agents logging in, but there may still be other Lync related issues, depending on what other services were impacted by the DC/AD reboot.  If this is done just to get things working quickly, then an after hours restart of servers in proper order (Lync, then prairieFyre) would be recommended to ensure no further unforeseen issues arise.

APPLIES TO

CCM for Lync 5.10