Service communications breaking due to bad clock synch

Not quite done with PS Remoting apparently, but I don't have time to fuss with it right now. Running remote commands now breaks the session, so that's not terribly helpful. Instead, I'm busy diagnosing this:

SearchServiceApplicationProxy::GetUserPreferenceSerializeHelper--Error occured: System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.

 ---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.

--- End of inner exception stack trace ---

Server stack trace:

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Office.Server.Search.Administration.ISearchSiteAdministrationServiceApplication.GetUserPreferenceSerializeHelper(Byte[] userID)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.<>c__DisplayClassf8.<GetUserPreferenceSerializeHelper>b__f7(ISearchServiceApplication serviceApplication)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName)

Internal server error exception: System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.

 ---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.

--- End of inner exception stack trace ---

Server stack trace:

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Office.Server.Search.Administration.ISearchSiteAdministrationServiceApplication.GetUserPreferenceSerializeHelper(Byte[] userID)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.<>c__DisplayClassf8.<GetUserPreferenceSerializeHelper>b__f7(ISearchServiceApplication serviceApplication)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, WebServiceBackedOperation`1 webServiceCall)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, WebServiceBackedOperation`1 webServiceCall)

at Microsoft.Office.Server.Search.Administration.UserPreference.GetUserPreference(Boolean lookupFromCache, SPWeb web)

at Microsoft.Office.Server.Search.Administration.UserPreference.GetUserPreference()

at Microsoft.Office.Server.Search.WebControls.SearchResultsBaseWebPart.ConfigureDataSourceProperties()

at Microsoft.Office.Server.Search.WebControls.SearchResultsBaseWebPart.EnsureWebpartReady()

System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.

---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.

--- End of inner exception stack trace ---    

Server stack trace:

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Office.Server.Search.Administration.ISearchSiteAdministrationServiceApplication.GetUserPreferenceSerializeHelper(Byte[] userID)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.<>c__DisplayClassf8.<GetUserPreferenceSerializeHelper>b__f7(ISearchServiceApplication serviceApplication)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, WebServiceBackedOperation`1 webServiceCall)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, WebServiceBackedOperation`1 webServiceCall)

at Microsoft.Office.Server.Search.Administration.UserPreference.GetUserPreference(Boolean lookupFromCache, SPWeb web)

at Microsoft.Office.Server.Search.Administration.UserPreference.GetUserPreference()

at Microsoft.Office.Server.Search.WebControls.SearchResultsBaseWebPart.ConfigureDataSourceProperties()

at Microsoft.Office.Server.Search.WebControls.SearchResultsBaseWebPart.EnsureWebpartReady()

Other blog postings have indicated this to be a time synchronization issue, and combing through the logs on the farm machines backs that up as a distinct possibility - I believe we are having some time synchronization fights going on between VMware ESX and the PDC, so now we just need to resolve the discrepancy and we shouldn't hear from this one again.

The infamous SP2010 User Profile Service

One of our contractor consultants is responsible with part-time administration tasks, and he assisted to a degree with the setup of our SP2010 environment. He struggled for some time with the configuration of the import of user profiles (we have five domains to pull profiles from currently), but last week he successfully got it configured and got profiles imported.

Thing is though, I was trying to look at some information about the profile import and anytime I'd open the Connections page, I would get the well-traveled "The query returns nothing" error.

After poring over his configuration for some time, I finally got tired of fussing with it and just nuked everything he did. Using the de facto guide published by Spencer Harbar (and some incremental fixing of Event Viewer log errors in trying to fix the original morass), I was able to successfully configure the User Profile Service Application in under 3 hours (had to add a NIC to the server in order to set up an SSL instance of MySites to save for later).

If you want something done right...