Complex .NET config transformations

In my previous post, I talked about (web).config transformations: how great they are and how they can be enhanced beyond the 'base install.' Continuing on, some of those enhancements later led us to some even more useful and advanced actions.

Using multiple configuration files in a .NET project is not the simplest thing to achieve. There are different mechanisms for including/merging files, but each have their limitations. One way is to specify that a section comes from another file, like so:

<connectionStrings configSource="otherfile.config"/>

The limitation with this approach is that the entire section must come from that other file. There is no 'merging' of elements; you couldn't add any additional elements to the <connectionStrings> section in this scenario.

You can get around this by importing the other config this way instead:

<appSettings file="otherfile.config">
   <add key="PagesToHide" value="AdminPage" />
   <add key="ExternalLinksToHide" value="Utilization Report" />
</appSettings>

The limitation with this approach is that this kind of import is not supported for all types of config sections:  <system.servicemodel>, for example, can't be used this way. (Also, some code inspection tools, like ReSharper, don't know how to parse this syntax.) And both these approaches force the imported file to be only that one config section; you couldn't have an <appSettings> section and a <connectionStrings> section in an external file and merge/import them both into your config.

In some scenarios, these limitations are not a problem. Both of these approaches have served us well (on a limited basis) in the past. Recently however we had the need to include a non-trivial set of configuration values into the config of multiple applications, most of which were already using web.config transformations. I generally assume that, as a developer, if I have copied and pasted something then I have failed. I wanted to find a more elegant, maintainable solution than just making each development team copy and paste the values into their individual base and transformation configs.

The solution we eventually came up involves a multi-step transformation. Don't be daunted by that "multi" though - it's actually quite a simple implementation.

We started by renaming the web.config (and its children web.debug.config & web.release.config) to web.base.config (children: web.base.debug.config & web.base.release.config). The content of these files we left untouched. We then added the files gateway.debug.config and gateway.release.config to the project. Each one looked something like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <appSettings>
    <add key="Thumbprint" 
         value="52CD92D192786742DA589FEA4C83719DA43E82C9" 
         xdt:Transform="Insert" />
    <add key="Version" 
         value="4" 
         xdt:Transform="Insert" />
    <add key="ConnectionString" 
         value="Data Source=db_host;Integrated Security=True" 
         xdt:Transform="Insert" />
  </appSettings>
  <system.serviceModel>
    <bindings>
      <basicHttpBinding>
        <binding name="BasicHttpBinding_Gateway"
                 closeTimeout="00:00:10" 
                 openTimeout="00:00:10" 
                 receiveTimeout="00:00:10" 
                 sendTimeout="00:00:10"
                 xdt:Transform="Insert">
          <security mode="Transport" />
        </binding>
      </basicHttpBinding>
      <wsHttpBinding>
        <binding name="WsHttpBinding_Gateway"
                 closeTimeout="00:00:10" 
                 openTimeout="00:00:10" 
                 receiveTimeout="00:00:10" 
                 sendTimeout="00:00:10"
                 xdt:Transform="Insert">
          <security mode="Transport">
            <transport clientCredentialType="Windows"
                       proxyCredentialType="None"
                       realm="" />
            <message clientCredentialType="Windows"
                     negotiateServiceCredential="true" />
          </security>
        </binding>
      </wsHttpBinding>
    </bindings>
    <client>
      <endpoint address="https://fake.url/GatewayService/4/a.svc/basic"
          binding="basicHttpBinding" 
          bindingConfiguration="BasicHttpBinding_Gateway"
          contract="namespace.interface"
          name="GatewayTransport_1"
          xdt:Transform="Insert" />
      <endpoint address="https://fake.url/GatewayService/4/a.svc/roles"
          binding="wsHttpBinding" 
          bindingConfiguration="WsHttpBinding_Gateway"
          contract="namespace.interface2"
          name="GatewayRoles_1" 
          xdt:Transform="Insert" />
    </client>
  </system.serviceModel>
</configuration>

The important parts to notice here are the xdt:Transform="Insert" attributes in each XML node. With XML transformations, it's possible to add (insert) elements as well as modifying and deleting them. (The xmlns attribute in the <configuration> element is also very important; without it Visual Studio doesn't know the file is a transformation document.)

The 'magic' comes in with a build step added to the project file:

<Target Name="BeforeBuild">
   <TransformXml 
       Source="web.base.config"
       Transform="Gateway.$(Configuration).config"
       Destination="obj\$(Configuration)\web.intermediate.config" />
   <TransformXml 
      Source="obj\$(Configuration)\web.intermediate.config"
      Transform="web.base.$(Configuration).config"
      Destination="web.config" />
</Target>

Now when the project builds, the compiler takes the base config and transforms it with the appropriate gateway config. Since the gateway config only has addition transformations, this effectively works like a merge of the two files. Then, the build configuration-specific transformation is performed, updating the elements that originally came from the base config to their build-appropriate values. This produces the web.config for the build configuration the solution was built in.

Like approaches discussed in the previous article, this solution is very seamless because it happens at compile-time, so you're able to validate the result at any point, and the web.config that IIS wants is always there. Plus, the gateway configs could be dropped into any project with ease because they just add on to what's already there. The one drawback is that you do have to rebuild to get IIS to pick up configuration file changes, whereas usually you can just save the file and refresh the web page. (This also makes the Visual Studio context menu item 'add config transform' not work, but then it doesn't always work anyway, and adding files to a project is a pretty trivial task.)

The example used here is for a web.config, but this should all work exactly the same with an app.config (so long as the build task is imported - see previous article.)

The possibilities presented by this technique are endless - you could do some pretty complex and powerful things by chaining transformation steps together. You don't want to have too many config files, but if you need to bring separation of concerns or just better readability to your application configuration, this is a powerful and elegant way to do it.

A simpler approach to .NET Config File Transformations

One of the nicer features of ASP.NET 4.0 is the concept of configuration file transformations. In case you are not familiar with it, here's a quick two-paragraph overview:

ASP.NET websites almost always need a different configuration for each deployment environment. The development/test configuration will differ in important ways from the staging configuration, and again from the production/live configuration. However, these differences are often only a smart part of the configuration file: a few URLs, server names and assorted settings. Before 4.0, the typical way to manage this was to maintain a separate, but 90% identical, web.config for each environment, and somehow make the right file the 'real' web.config at deployment. Keeping the files in sync was not a trivial task, especially with large projects. We had one application with a config that was well over a thousand lines, and the various environment 'editions' have gotten so out of sync with each other that we spent an embarrassing amount of time squashing bugs that would only manifest in one environment (which of course was usually production).

Web.config transformations brings some relief to this problem. With this feature, you have the web.config, and then it has several child files, one for each build configuration. The primary web.config contains all the information, and then each child file contains instructions for modifications. For example, the primary web.config might contain a section defining the test database connection string, and then the child file web.release.config contains an instruction to replace that section with the production database connection string. When you publish the website, it will transform the primary web.config based on the child config file that corresponds to the build configuration being published, resulting in a file that is valid for the environment. (These 'instructions' are an XML transformation language developed by Microsoft.)

As useful as this feature is, there are some difficulties.

The main difficulty is that transformations are not created at build time. This can make them a little hard to validate, because you have to go out and use a secondary tool or script to execute the transforms and check them over. You can do this from the command-line like so:

msbuild /nologo /target:TransformWebConfig /p:Configuration=Release IISHost.csproj

This will create the the transformed config (for the Release build configuration) in obj\Release\TransformWebConfig\transformed\Web.config.

In the spirit of making this a usual part of the build process, we attempted to include this as a post-build event. However, because this MSBUILD task actually compiles the project, this creates a circular dependency where the project is built, and then the post-build event kicks off, which builds the project a second time, which kicks off the post-build event, which keeps looping back on itself infinitely until your computer crashes.

Not generating the transforms at build-time also causes issues for deployment. Transforms are generated when a website is published, but the Publish mechanism is not the best or preferred way to deploy. In enterprise environments especially, the deployment team typically does not have development tools such as Visual Studio or MSBUILD installed, and generally just wants to have a folder of files they can copy to the correct directory. (Publish also does not lend itself to managing backups, versioning, or rollbacks.) So build engineers and deployment teams have to manually execute the transform, and then go through and copy and rename files just as they did before ASP.NET 4.0.

The third issue is that it's also only available for web.config files, not for app.config files. Now, there are plug-ins that add this feature for non-web projects, such as SlowCheetah. SlowCheetah is a good solution, but I was always hopeful I could find a simpler, more integrated approach. A number of articles, especially this one, told me it was possible, but their solutions were more complex than I really wanted. They did however guide me to this conclusion:

It is possible to execute the transform as a native part of the build by adding some elements to the project file. These elements cannot be added through the Visual Studio GUI (as far as I know), so you do have to manually edit the project file, but these additions are very small, simple bits of XML, so it's not too daunting.

In a website project, at the bottom of the file (just before the </Project> element), add the following:

<Target Name="AfterBuild">
    <TransformXml Source="web.config"
                  Transform="web.$(Configuration).config"
                  Destination="obj\$(Configuration)\web.config" />
</Target>

The target name is very important; that's the part that tells the compiler to run the TransformXml task as an after-compile task. (Because it is not making an external, recursive call, the afore-mentioned infinite loop condition is not created.) Once this section is added, every build will produce, in the obj folder corresponding to the build configuration just compiled, the transformed web.config! The beauty of this solution is that the transformed config is always available when you want to validate it (or when build engineers or deployment teams need to copy it) but the default, development web.config remains in place and untouched.

TransformXml is included by default for website projects. You can do the same thing for app.config files, but you have to explicitly import the task in non-web project files. You can do this by including the following line before the <Target> element:

<UsingTask 

   TaskName="TransformXml" 

   AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v$(VisualStudioVersion)\Web\Microsoft.Web.Publishing.Tasks.dll" 

/>

(The DLL referenced here is included as part of the Visual Studio install, so you shouldn't have to add any packages or features. It does not need to be included in the Project References either.)

Being able to automate the configs in this manner has been very helpful for us. In my next post, I'll go into detail about how we were able to leverage this knowledge to accomplish even more advanced tasks.

EDIT: If you have other post-build tasks that rely on the transformed web.config, you should use "BeforeBuild" as the target name. There appears to be a slight delay between the build finishing and the final file being written out. To avoid any race conditions, use "BeforeBuild" instead of "AfterBuild".)

EDIT 2 (7/15/2016): I have modified the <UsingTask> path to use $(VisualStudioVersion) so the project file is more maintainable.