Converting from MSTest to NUnit

Ever since I was converted to the one true gospel of Test-Driven Development in 2008, I have had to convert test projects from MSTest to NUnit and back a few different times. After my last time having to do it, I decided it was time to testify.

The Facts

MSTest and NUnit have different names for the decorations on the test classes & methods:

MSTest Attribute	NUnit Attribute	Purpose
[TestMethod]	[Test]	Identifies of an individual unit test
[TestClass]	[TestFixture]	Identifies of a group of unit tests, all Tests, and Initializations/Clean Ups must appear after this declaration
[ClassInitialize]	[TestFixtureSetUp]	Identifies a method which should be called a single time prior to executing any test in the Test Class/Test Fixture
[ClassCleanup]	[TestFixtureTearDown]	Identifies a method in to be called a single time following the execution of the last test in a TestClass/TestFixture
[TestInitialize]	[SetUp]	Identifies a method to be executed each time before a TestMethod/Test is executed
[TestCleanUp]	[TearDown]	Identifies a method to be executed each time after a TestMethod/Test has executed
[AssemblyInitialize]	N/A	Identifies a method to be called a single time upon before running any tests in a Test Assembly
[AssemblyCleanUp]	N/A	Identifies a method to be called a single time upon after running all tests in a Test Assembly

Source: Comparing the MSTest and Nunit Frameworks, by Naysawn Naderi, 1 Feb 2007.
This article contains some information that is no longer accurate, particularly in regards to the NUnit test runner, but this chart is still correct.

There are a few other minor syntactical differences to be aware of:

1. Methods decorated with [ClassInitialize] in MSTest must have the signature
   public static void MethodName(TestContext context)
   while methods decorated with [TestFixtureSetUp] cannot have any arguments.
2. MSTest's Assert and NUnit's Assert both have a method IsInstanceOfType(), but the order of parameters is reversed between frameworks.
   MSTest: void IsInstanceOfType(object actual, Type expectedType)
   NUnit:    void IsInstanceOfType(Type expected, object actual)

The Editorial

The debates over whether to use MSTest or NUnit typically revolve around three factors:

1. Speed
2. Integration
3. Ease of use

People are pretty evenly split as to which one is actually faster. Personally I've found that so many factors affect this (the version of Visual Studio being used, the complexity of the tests, the framework being targeted, the version of NUnit) that we have to just call this one a tie.

When it comes to integration, if you're using Resharper, then it's a moot point. Resharper can run MSTest and NUnit tests in the very same test run seamlessly, and with a far superior interface to MSTest. And if you're not using Resharper, well then I just don't know how you do it. I had to go without Resharper for a week at a new job once, and it was like trying to run in waist-deep water. Everything took five times as long, and my confidence in my code was significantly reduced.

MSTest requires at least twice as many keystrokes to write tests than NUnit - everything is more verbose, with no additional benefit (see the difference in method signatures above for a minor example). Plus, NUnit can do this:

[TestCase("ConnectionStringA")]
[TestCase("ConnectionStringB")]
public void CreatePendingPaymentTest(string connectionName)
{
 //setup

 Repository classUnderTest = new Repository(connectionName);
 classUnderTest.CreateRecord();
 
 //assert
}

[TestCase] allows you to run the same test with different settings/inputs.
You can accomplish the same thing with MSTest by writing multiple tests that call the same method with different parameters, but again, more work to get the same result.

So, with speed being equal, integration being a non-issue, and NUnit the easier to use, the verdict is clear: use NUnit. It's available through NuGet when you need to add it into your solution, and you can learn more at NUnit.org. Enjoy!

EDIT (6/18/2015):
The syntactical differences section omitted an item that is now included.

An evaluation of Silverlight and XAML

Up till now, I've only used this blog to post about technical issues and development patterns, without really editorializing. This entry, however, is going to be a rather subjective evaluation of technology stacks.

One of the bigger paradigm shifts in .NET development that has occurred in the last few years is the introduction of Silverlight. I'm not going to go into the reasons that Silverlight was introduced or why it has seen a significant rate of adoption (partly because the reasons for the latter are incredibly varied depending on company and application). Hand in hand with Silverlight has come XAML. The two are not one and the same: Silverlight is an application framework that's very tightly coupled to Internet solutions, while XAML is a markup language that can be used in many different types of .NET projects. This post will discuss the pros and cons of both.

Silverlight is nice in that it further simplifies web application programming. Some websites are actually not suited to a model where the user can control page flow, and some user interfaces can only be accomplished in an HTML+JavaScript environment after much trickery and tweaking third-party controls (e.g. jQuery UI). Silverlight allows you to deliver an application in a sever-client and/or web-based manner without having to dance around the stateless, static-content-oriented HTTP protocol.

Silverlight's architecture has some good ideas behind it. Applets have been a source of (real and perceived) security concerns for a long time, so the Silverlight designers decided that Silverlight would be a subset, instead of a super-set, of the .NET framework. In other words, Silverlight uses a smaller selection of .NET's functionality. This deliberate scope restriction takes away the ability of Silverlight applets to do some dangerous things. The code to access local file systems, databases, and other critical resources is not even there. It enforces a safer application ecosystem by design instead of by potentially breakable (and in the end arbitrary) access switches. It also solves (or at least mitigates) another concern of applets. Applets have to have their own execution sandbox that the client has to download, in addition to downloading the applet itself. A smaller functionality set compiles to smaller binaries, which results in smaller download and install footprints. Even with today's fast connections, dual-core processors and hundred-gig hard drives, all resources are still finite, and so creating a paradigm that rewards lighter-weight deliverables is a very smart idea.

These benefits do not come without some significant hassles, though.

One of the biggest problems I've had with Silverlight is that various versions are not compatible. You can install and run .NET 1.1, 2.0, and 4.0 all on the same machine without any problem - you can have solutions that have projects in all different versions; you can have multiple IIS app pools running different framework versions all running simultaneously. Not only is this possible, but it's extremely easy - the setup and execution is all seamless. .NET is certainly not the only application framework or product that can support this kind of parallelism, but the point is that it does it well. Silverlight does not. I know it is possible to get Silverlight 3 and 4 running on the same machine - I've seen co-workers get it done - but its a very difficult process, and even those co-workers throw up their hands in surrender when I ask them to help me re-create it. "I just un-installed and re-installed things in an apparently arbitrary order until it started working" was the answer I got from more than one of them.

On the surface this sounds like a nit-picky concern - "just use the same version of Silverlight for everything", right? But let's be realistic, it's never that simple. Various applications are developed under different constraints and requirements, and sometimes using only one version is simply not a realistic option. Some clients and environments require an older framework, and you can't change that. Plus, even if you do have the option to upgrade, development hours are limited and business users/clients aren't always willing to assume the upgrade risk. This is true for anything, not just Silverlight - there are still many .NET 2.0 applications and DLLs running in production environments that won't be upgraded for years to come for these very same reasons. Effective multi-version support is a feature I don't think enterprise software development tools can skimp on, and I feel that Silverlight not only skimped, but completely dropped the ball.

This versioning/parallelism flaw is major, but there are also some important minor annoyances. The subset mentality that Silverlight was designed in is a good idea that was clumsily executed. Instead of Silverlight being a 'true' subset of .NET, it is actually a parallel, minimized fork of .NET. It looks like .NET, it smells like .NET, but it doesn't taste like .NET. Visual Studio is always cranky when you try to add a reference to a Silverlight project in a non-Silverlight project - it'll do it, and the solution will compile, but VS will always mark it as a broken reference in Solution Explorer. Tools like ReSharper will even give you pre-compile errors in non-Silverlight code that references Silverlight code (as well as in the solution-wide analysis, which is much harder to ignore).

The path of work-arounds this particular flaw sent me down was a real comedy of errors. "Hmm, VS2010 + Resharper doesn't play nice with MSTest projects referencing Silverlight projects. Okay, let's create a Silverlight test project. Hmm, no such project type. Okay, we'll just create a Silverlight class library - the whole 'test project' definition is a somewhat arbitrary distinction anyway. Argh, okay, where can I download the MSTest for Silverlight framework? Geez that's hard to find. Okay got it! Compiles, woo-hoo! What? The MS Test runner can't run the MSTest for Silverlight tests (doesn't recognize the attributes)? Crap. Well, maybe there's a port of the test runner for Silverlight. Uh ... well there's a crappy browser-based version that's difficult to use and hard to see test failures in... Okay, screw it, let's just go with NUnit, I've never really liked MSTest anyway. NUnit port for Silverlight? Unofficially done, but existent and stable! Score one for open-source! Create Silverlight class library, add tests, reference NUnit for Silverlight DLLs ... compiles! Woo-hoo! RUNS! Woo-hoo!"

I share that bit partly to inject a little levity, partly to show that with Silverlight NUnit is nicer than MSTest, and partly to reinforce my argument that Silverlight is a second-class citizen even in the Microsoft world. MSTest doesn't like it, Visual Studio doesn't like it, it doesn't even like itself. There are just so many little 'gotchas' in trying to use Silverlight, functionality that has to be re-created, or specialized ports of existing tools that you have to employ. The whole paradigm just seems to work against code re-use, which is something that makes me rather cranky.

XAML is the markup language that Silverlight uses to create its user-interface components. As mentioned previously, though, it is not tied to Silveright. The Windows Presentation Foundation (WPF), which is intended for desktop applications, also uses XAML. In fact, Visual Studio 2010 itself is written in WPF, and therefore XAML. XAML is a big leap forward in terms of simplicity and portability of UI design - it takes everything that was great about HTML, CSS, and Web Forms, and combines them all into something even better. It further closes the gap between Windows Forms and Web Forms - these two technologies used extremely similar but inherently separate structures, but now everything is united under one roof. You can design for the desktop or the web (as long as that web is Silverlight) using one approach. XAML makes formatting pages/screens much, much more intuitive than setting up CSS stylesheets or creating application themes, and it makes the flexibility of HTML layouts available to desktop apps. Making a desktop application look pretty is no small feat regardless of technology, and WPF gives you a shorter path.

Unfortunately, XAML also takes everything that was bad about ASP.NET Data Grids and makes it the standard. The Model-View Model pattern that XAML is intended to employ encourages injecting property, method, and even class names directly in the XAML markup, or in other words, into uncompiled text. I shudder every time I see this kind of thing being done, whether it's in .config documents, vanilla XML, or in 'magic' strings inside the C#/VB code. Doing this kind of thing works against refactoring. As far as I'm aware, there exists no tool that will extend object refactors into the XAML. Given the fluid nature of the XAML data-binding model, it's a difficult task to hope to accomplish, especially considering the fact that the source object doesn't have to be bound in until run-time. Again, this may seem nit-picky, but I argue that it is not. Code is always changing, and needs to be flexible enough to accommodate rapid change. This need becomes more and more pressing each year. Members in XAML {Binding} or {StaticResources} statements are disconnected from the code in a way that discourages and complicates changes. What's even more concerning to me is that it is very easy for incomplete refactors to go unnoticed. It is very easy to change something, have the {Binding} member no longer match, and then that element no longer shows up on the screen, and no one would even notice, even with the greatest QA department in the world, because no error is thrown when said binding fails. This kind of thing has bitten us more than a few times even with the more strict binding mechanism of ASP.NET Data Grids, sometimes even in production code. I am pessimistic that such occurrences will only increase in a world that relies more heavily on XAML-based implementations.

Now, the good news is that there are ways to get around this flaw in XAML. The traditional, explicit data-binding model of giving controls names and wiring them up in the code-behind can be employed. There are also code-only ways to create {Binding}s using only C# code (no XAML) - they're not as pretty, but they work, and because they eschew magic-string based reflection they are refactor-friendly. I hope to post some examples of my own here before too long.

My current evaluation of Silverlight is that it has too many flaws to justify the somewhat dubious benefits it brings. In the end, traditional ASP.NET websites with a liberal amount of jQuery can provide all the same functionality without any of Silverlight's limitations or contrivances. And if the HTML 5 standard can ever see wide-spread adoption, then Silverlight becomes even less attractive. I would urge .NET developers to discourage the use of Silverlight in order to shorten the time till its end-of-life date.

My current evaluation of XAML (and I reserve the right to modify this in the future) is that it is better than both Windows Forms and Web Forms. While it has some non-trivial pitfalls, they are worth the risk for the benefits gained. I would urge the use of WPF for desktop development, and if it ever becomes available for non-Silverlight ASP.NET use, then it is preferable to 'pure' HTML+CSS.

Comments, questions, and corrections are more than welcome!

MS Test, Private Accessors, and Team Build

At my work,we use TeamBuild to run a continuous integration build on a couple of different solutions.  Recently, we had an issue where one test project (there are several in any given solution) would run its tests just fine in Visual Studio on the developers' machines, but when run under the command-line MSTest runner for TeamBuild, they would fail.  These tests were using a private accessor, and the test failures were reporting that the MSTest run-time could not properly load the class the private accessor was trying to wrap.  (Private Accessors use reflection to wrap the classes they give you access to, and it was giving a "Could not load file or assembly" error.)

I spent the better part of two days researching this issue.  I could never reproduce it on my workstation, and the particularly bizarre thing was that the tests would run fine through the Visual Studio instance installed on the TeamBuild machine.  Something was off with the way MSTest was doing things, it seemed.  I never found anything on the Internet that really described this issue, though I did find some similiar issues relating to TeamBuild and reflection.  I tried a couple of thier solutions, as well as tweaking config and rewriting the tests a bit.  Nothing worked.

Eventually, I just threw in the towel, and wrote my own private accessor class, and that finally resolved the issue.  Here's what I did:

The class were trying to test was called PageTitleModule, and a private method that needed to be tested, which depended on the state of a private variable.  Inside the test class, there was a place where we built the PageTitleModule_Accessor class and set up the private variable:

private static PageTitleModule_Accessor GetPageTitleModule(PageTitleInfo cmsContent)
{
   PageTitleModule_Accessor module = new PageTitleModule_Accessor();
   module._cmsContent = cmsContent;
   return module;
}

Then, in the actual tests, we would use the accessor class to invoke the private method:

PageTitleModule_Accessor module = GetPageTitleModule(cmsContent);
string reWrittenResponse = module.RewriteResponse(originalResponse);

Just for reference, we use this same pattern, including the built-in private accessors, many places in our test code, and none of them ever given us any grief.  So I'm sure there is some setting that was wrong or compilation option that needed to be tweaked.  But after two days, the hack-y workaround I'm about to show you just made more sense; we needed to get our tests running again so we could get back to actual development.

Before, when we had the problem, we were using the Private Accessor fuctionality built-in to .NET.  (In other words, we went to PageTitleModule, right-clicked on the class definition, told it to "Create Private Accessor" and selected the appropriate test project, thus generating the PageTitleModule_Accessor class.)  When I finally gave up on that working, I wrote a PageTitleModule_Accessor of my own that implemented this private member and private method:

internal class PageTitleModule_Accessor
{
  private readonly PrivateObject _moduleImpl;

  public PageTitleModule_Accessor()
  {
    _moduleImpl = new PrivateObject(typeof(PageTitleModule), null);
  }

  public PageTitleInfo _cmsContent
  {
    get
    {
      return _moduleImpl.GetFieldOrProperty("_cmsContent") as PageTitleInfo;
    }
    set
    {
      _moduleImpl.SetFieldOrProperty("_cmsContent", value);
    }
  }

  public string RewriteResponse(string response)
  {
    return _moduleImpl.Invoke("RewriteResponse", new object[] {response}) as string;
  }
}

This PrivateObject type that I'm using is part of the Microsoft's unit testing framework (Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll).  While researching why the default accessor generation was not working, I disassmbled the generated test assembly (using RedGate's .NET Reflector) to take a look at what was going on and where the error was coming from.  I didn't find what I was looking for, but I saw that the auto-generated private accessors employ this PrivateObject type to manage the reflection of the type being tested, so I elected to re-use it.

(The "generated test assembly" mentioned above refers to the fact that when you use the built-in private accessor functionality, when you build, the C# compiler creates a DLL named [AssemblyUnderTest]_Accessor.dll, and it is in this binary that the [ClassUnderTest]_Accessor type lives.)

So, for those of you who know a lot about .NET private accessors, I'd love to have you weigh in on this.  Those of you who might happen to run into this problem, hopefully this gives you some information on what to do with it.