In my daily work I am often called in as a “terminator” for tough problems that cannot be resolved through normal support channels. Not long ago I was tasked to figure out why a custom SharePoint 2010 Feature was failing to activate on a production farm. Using SharePoint Features [microsoft.com] is an awesome way to package and deploy custom code in a controlled manner. However in this case, the following error was being generated on activation:

Error. An object of the type Microsoft.SharePoint.Administration.SPWebConfigJobDefinition named “job-webconfig-modification” already exists under the parent Microsoft.SharePoint.Administration.SPWebService named “”. Rename your object or delete the existing object.

The error only occured in production and not in the dev and stage environments. So, after checking there was nothing obviously wrong with the code, the original programmer thought it must be an issue with the SharePoint 2010 production farm itself. Although everything seemed to be performing just fine, just to be safe, our SharePoint administrator ran diagnostics and clean up routines and even had Microsoft support check things out. Everything was confirmed to be in working order and yet the feature still failed consistently with the same error leaving the server configuation in an inconsistent state.

The custom feature in question was designed to add web.config settings to the SharePoint 2010 farm programmatically. Microsoft documents how to achieve this sort of modification in an MSDN article titled How to: Add and Remove Web.config Settings Programmatically [microsoft.com]. Here is the relevent C# code from that article:

SPWebService service = SPWebService.ContentService;

SPWebConfigModification myModification = new SPWebConfigModification();
myModification.Path = "configuration/SharePoint/SafeControls";
myModification.Name = "SafeControl[@Assembly='MyCustomAssembly'][@Namespace='MyCustomNamespace'][@TypeName='*'][@Safe='True']";
myModification.Sequence = 0;
myModification.Owner = "User Name";
myModification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
myModification.Value = "";
service.WebConfigModifications.Add(myModification);
 
/*Call Update and ApplyWebConfigModifications to save changes*/ 
service.Update();
service.ApplyWebConfigModifications();

Pretty straightfoward in a SharePoint-y kind of way. However, when I inspected the source code for the custom feature I found the modifications were being made using a “helper” function to simplify the syntax. So the code looked something like this:

HelperToAddWebConfigModification("SafeControl[@Assembly='MyCustomAssembly1'][@Namespace='MyCustomNamespace'][@TypeName='*'][@Safe='True']");
HelperToAddWebConfigModification("SafeControl[@Assembly='MyCustomAssembly2'][@Namespace='MyCustomNamespace'][@TypeName='*'][@Safe='True']");
HelperToAddWebConfigModification("SafeControl[@Assembly='MyCustomAssembly3'][@Namespace='MyCustomNamespace'][@TypeName='*'][@Safe='True']");

Note that this is greatly simplified and, like all the examples on this site, it is not taken from any actual production code. In this case the example is meant to demonstrate that the helper function was committing individual web.config modification one at a time instead of calling service.Update() and service.ApplyWebConfigModifications() once for the batch of changes.

It turns out that calling service.Update() and service.ApplyWebConfigModifications() was causing a timing issue on our multi-server production farm which did not present itself in single server development and staging farms. Rewriting the code to match the MSDN style and removing the “helper” functions so all three updates could be made a single call to service.Update() and service.ApplyWebConfigModifications() resolved the issue nicely.

There are a few take away lessons from this incident. First off, if our staging farm had been multi-server to more closely match the production farm we could have caught the issue before a production deployment. Luckily, the deployment process had a number of other safety nets in place so the error did have any impact our users. Hooray for “defense in depth” 🙂

Secondly, although the error message could plausibly be interpreted as a server configuration issue, as is often the case, the real cause was not easily infered from the error alone. After checking out the basic configuration, if I had not stepped back to look at the bigger picture we could have spent a lot more time having the server administration team track down a “ghost” process.

Finally, and most importantly in my mind, this is a cautionary tale about rewriting basic functionality. In trying to simplify awkward syntax and APIs, it is all to easy to introduce unintended side effects in these types of helper functions unless you take a strict refactoring approach. This is especially true in SharePoint. These type of helper functions also make things harder to debug because instead of looking at the standard “ugly” syntax everyone is used to seeing, it is just one more layer of an onion that needs to be peeled back to get to the root of the problem.