Service communications breaking due to bad clock synch

Not quite done with PS Remoting apparently, but I don't have time to fuss with it right now. Running remote commands now breaks the session, so that's not terribly helpful. Instead, I'm busy diagnosing this:

SearchServiceApplicationProxy::GetUserPreferenceSerializeHelper--Error occured: System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.

 ---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.

--- End of inner exception stack trace ---

Server stack trace:

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Office.Server.Search.Administration.ISearchSiteAdministrationServiceApplication.GetUserPreferenceSerializeHelper(Byte[] userID)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.<>c__DisplayClassf8.<GetUserPreferenceSerializeHelper>b__f7(ISearchServiceApplication serviceApplication)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName)

Internal server error exception: System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.

 ---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.

--- End of inner exception stack trace ---

Server stack trace:

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Office.Server.Search.Administration.ISearchSiteAdministrationServiceApplication.GetUserPreferenceSerializeHelper(Byte[] userID)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.<>c__DisplayClassf8.<GetUserPreferenceSerializeHelper>b__f7(ISearchServiceApplication serviceApplication)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, WebServiceBackedOperation`1 webServiceCall)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, WebServiceBackedOperation`1 webServiceCall)

at Microsoft.Office.Server.Search.Administration.UserPreference.GetUserPreference(Boolean lookupFromCache, SPWeb web)

at Microsoft.Office.Server.Search.Administration.UserPreference.GetUserPreference()

at Microsoft.Office.Server.Search.WebControls.SearchResultsBaseWebPart.ConfigureDataSourceProperties()

at Microsoft.Office.Server.Search.WebControls.SearchResultsBaseWebPart.EnsureWebpartReady()

System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.

---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.

--- End of inner exception stack trace ---    

Server stack trace:

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)

at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Office.Server.Search.Administration.ISearchSiteAdministrationServiceApplication.GetUserPreferenceSerializeHelper(Byte[] userID)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.<>c__DisplayClassf8.<GetUserPreferenceSerializeHelper>b__f7(ISearchServiceApplication serviceApplication)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, WebServiceBackedOperation`1 webServiceCall)

at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, WebServiceBackedOperation`1 webServiceCall)

at Microsoft.Office.Server.Search.Administration.UserPreference.GetUserPreference(Boolean lookupFromCache, SPWeb web)

at Microsoft.Office.Server.Search.Administration.UserPreference.GetUserPreference()

at Microsoft.Office.Server.Search.WebControls.SearchResultsBaseWebPart.ConfigureDataSourceProperties()

at Microsoft.Office.Server.Search.WebControls.SearchResultsBaseWebPart.EnsureWebpartReady()

Other blog postings have indicated this to be a time synchronization issue, and combing through the logs on the farm machines backs that up as a distinct possibility - I believe we are having some time synchronization fights going on between VMware ESX and the PDC, so now we just need to resolve the discrepancy and we shouldn't hear from this one again.