Tuesday, January 18, 2011

Unexpected Response: System.ServiceModel.ServiceActivationException

Had a SharePoint site migrated from MOSS 2007 to SharePoint 2010 via the content DB detach/re-attach method. Not sure that was a factor here, but adding this little fact for posterity.

Problemo:

Editing a publishing page produced the following popup dialog in IE, FireFox, Chrome, etc when a user attempted to edit a publishing page that was already checked out by another user:

          image

“Unexpected response from server. The status code of the response is ‘500’. The status text of response is ‘System.ServiceModel.ServiceActivationException”

Investigation into the server’s event log showed this helpful tidbit of information:

WebHost failed to process a request.
Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/2593506
Exception: System.ServiceModel.ServiceActivationException: The service '/_vti_bin/client.svc' cannot be activated due to an exception during compilation.  The exception message is: Operation is not valid due to the current state of the object.. ---> System.InvalidOperationException: Operation is not valid due to the current state of the object.
   at Microsoft.SharePoint.ApplicationRuntime.SPLayoutsMappedFile.MapLayoutsVirtualPath(String virtualPath)
   at Microsoft.SharePoint.ApplicationRuntime.SPVirtualPathProvider.GetCacheKey(String virtualPath)
   at Microsoft.SharePoint.Publishing.Internal.CmsVirtualPathProvider.GetCacheKey(String virtualPath)
   at System.Web.Compilation.BuildManager.GetCacheKeyFromVirtualPath(VirtualPath virtualPath, Boolean& keyFromVPP)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile)
   at System.Web.Compilation.BuildManager.GetCompiledCustomString(String virtualPath)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.GetCompiledCustomString(String normalizedVirtualPath)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
   --- End of inner exception stack trace ---
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
   at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath)
Process Name: w3wp

Yet more evidence to be used in a future blog post: Why I hate WCF!

Solution-o:

So searching for this issue produces a few results – though none that had the exact ‘Operation is not valid due to the current state of the object’ wording. The results that did come back were ones that were all too familiar when I had made the mistake of using WCF in the past: cannot have multiple bindings, multiple authentication mechanisms, etc. With the solutions generally being along the lines of creating an extended site. Oh boy – another IIS site to manage! 

The deal with this particular site though did not fall in line with those solutions – it only had one binding (single AAM to 443) – though it was anonymous enabled (but not claims-enabled).

So how did I fix? Here are the steps:

  1. Travel to C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\ISAPI on each WFE – or wherever your 14-Hive exists.
  2. Tip: Make a backup copy of the web.config file in that folder. You just never know…
  3. Edit the web.config file by adding the highlighted line to enable aspNetCompatibility:

<configuration>
  <system.web>
      …
     
<serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
  </system.serviceModel>
</configuration>

Save your changes – that should be all you need to do. App pool will recycle…

Why does this work? Who the heck knows – ask the WCF team – you may find them working on the Office Assistant 2.0 project. My guess is that things work better (but maybe slower? more limitedly?) when the ASP.NET pipeline joins the party. Will post here if there are any side effects – so far so good though. 

11 comments:

  1. Interesting fix. Worked for me, but I agree, it's a weird fix. Service was working fine one day and then broken the next with no apparent reason.

    Also, clever blog sub-title ;)

    ReplyDelete
    Replies
    1. Thanks, C-Digital. Glad fix worked. Some day I will get around to publishing my WCF Hatred Manifesto...

      Delete
  2. Thanks! Thanks! Thanks! This issue was driving me crazy.... I have weeks digging into this. I am surprised the solution were so simple and so difficult to at least for to find

    ReplyDelete
  3. Awesome! Glad to help.

    WCF is foiled again (in its mission to break sites and cause headaches)!

    ReplyDelete
  4. worked for me. Thanks for posting.

    Utsab

    ReplyDelete
  5. I'm having this issue with SP 2013 site. I get the error when I navigate to Term-Driven pages in Term Store. Also when I edit a Content Query Search WP, in the results page same error shows up.
    I added the attribute you mentioned into web.config - the one under root of 15\ISAPI folder, however after doing that Term Store page becomes blank and there is a warning in the even viewer saying:

    It is an error to use a section registered as allowDefinition='MachineToApplication' beyond application level. This error can be caused by a virtual directory not being configured as an application in IIS. (C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\isapi\web.config line 243)

    Also tried extending the web site, no luck! What am I doing wrong here? Do I need enter this attribute to another web.config?

    ReplyDelete
    Replies
    1. So sorry - first time I have heard about this happening on 2013. Real bummer.

      I think this is purely an IIS + ASP.NET problem, so no need to go the route of extending your site in order to address this issue.

      I was unable to duplicate your issue in my local 2013 farm by making the same change to the web.config at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\isapi\web.config

      Here is the last few lines of my web.config:









      That message usually comes up when a web.config section has been 'locked' by a parent web.config file where it is defined as now allowing web.config changes to this particular section. Has your SharePoint farm been through any type of server hardening? Your error indicates that the 'system.serviceModel' is locked, since that is the section being modified here - which is certainly strange.

      Here is how I might troubleshoot this:

      1) Get to know which .config files are involved here. Your web.config settings are basically inherited from other .config files on the server. This msdn link should be a good start: http://msdn.microsoft.com/en-us/library/ms178685.aspx

      2) Hunt through each file in the hierarchy, searching for "allowDefinition" on a "system.serviceModel" section.

      3) If found, change the attribute to 'Everywhere' (after backing up the file you are changing), save, IISRESET, then test results.


      Hopefully this does not take you too long to try out, since even if you do mange to find the right file + section to change, you still may not have found a fix to your original issue. And then there is the question about why the section was set to 'MachineToApplication' in the first place...

      Delete
    2. My web.config was stripped out - here it is:

      <dataContractSerializer maxItemsInObjectGraph="2147483647" />
      </behavior>
      </endpointBehaviors>
      </behaviors>
      <serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
      </system.serviceModel>
      </configuration>

      Delete
  6. I am having this issue as intermittent , sometime this issue comes up stays for a week and so and the automatically disappear , even i tried to change the config file but no success was achieved.

    What wrong is happening here?

    ReplyDelete
    Replies
    1. Next time you have the issue, take a look at what is happening on the server. Is the CPU maxed? Is the w3wp process using an inordinately high amount of memory? WCF seems to adhere to the 'we will sacrifice ourselves rather than kill the server' policy - which is mostly good. I have this server that barely meets minimum specs for SharePoint and has WCF activation issues after a duration of uptime. Requires an IISRESET to clear the way for successful WCF operations - until it happens again.

      So here I would start with:
      - Look at server status, as described above
      - See if app pool recycle clears the issue next time it happens. If yes, then you have mem leak to deal with. Update to latest Svc Packs and audit custom solutions.
      - If you cannot get to the bottom of the leak, try increasing the frequency of app pool recycles (during low usage periods of course) to see if the problem becomes less frequent. If this helps, consider beefing up your server's memory capacity.

      Delete

 
© I caught you a delicious bass.
Back to top