Converting from MSTest to NUnit

Ever since I was converted to the one true gospel of Test-Driven Development in 2008, I have had to convert test projects from MSTest to NUnit and back a few different times. After my last time having to do it, I decided it was time to testify.

The Facts

MSTest and NUnit have different names for the decorations on the test classes & methods:

MSTest Attribute	NUnit Attribute	Purpose
[TestMethod]	[Test]	Identifies of an individual unit test
[TestClass]	[TestFixture]	Identifies of a group of unit tests, all Tests, and Initializations/Clean Ups must appear after this declaration
[ClassInitialize]	[TestFixtureSetUp]	Identifies a method which should be called a single time prior to executing any test in the Test Class/Test Fixture
[ClassCleanup]	[TestFixtureTearDown]	Identifies a method in to be called a single time following the execution of the last test in a TestClass/TestFixture
[TestInitialize]	[SetUp]	Identifies a method to be executed each time before a TestMethod/Test is executed
[TestCleanUp]	[TearDown]	Identifies a method to be executed each time after a TestMethod/Test has executed
[AssemblyInitialize]	N/A	Identifies a method to be called a single time upon before running any tests in a Test Assembly
[AssemblyCleanUp]	N/A	Identifies a method to be called a single time upon after running all tests in a Test Assembly

Source: Comparing the MSTest and Nunit Frameworks, by Naysawn Naderi, 1 Feb 2007.
This article contains some information that is no longer accurate, particularly in regards to the NUnit test runner, but this chart is still correct.

There are a few other minor syntactical differences to be aware of:

1. Methods decorated with [ClassInitialize] in MSTest must have the signature
   public static void MethodName(TestContext context)
   while methods decorated with [TestFixtureSetUp] cannot have any arguments.
2. MSTest's Assert and NUnit's Assert both have a method IsInstanceOfType(), but the order of parameters is reversed between frameworks.
   MSTest: void IsInstanceOfType(object actual, Type expectedType)
   NUnit:    void IsInstanceOfType(Type expected, object actual)

The Editorial

The debates over whether to use MSTest or NUnit typically revolve around three factors:

1. Speed
2. Integration
3. Ease of use

People are pretty evenly split as to which one is actually faster. Personally I've found that so many factors affect this (the version of Visual Studio being used, the complexity of the tests, the framework being targeted, the version of NUnit) that we have to just call this one a tie.

When it comes to integration, if you're using Resharper, then it's a moot point. Resharper can run MSTest and NUnit tests in the very same test run seamlessly, and with a far superior interface to MSTest. And if you're not using Resharper, well then I just don't know how you do it. I had to go without Resharper for a week at a new job once, and it was like trying to run in waist-deep water. Everything took five times as long, and my confidence in my code was significantly reduced.

MSTest requires at least twice as many keystrokes to write tests than NUnit - everything is more verbose, with no additional benefit (see the difference in method signatures above for a minor example). Plus, NUnit can do this:

[TestCase("ConnectionStringA")]
[TestCase("ConnectionStringB")]
public void CreatePendingPaymentTest(string connectionName)
{
 //setup

 Repository classUnderTest = new Repository(connectionName);
 classUnderTest.CreateRecord();
 
 //assert
}

[TestCase] allows you to run the same test with different settings/inputs.
You can accomplish the same thing with MSTest by writing multiple tests that call the same method with different parameters, but again, more work to get the same result.

So, with speed being equal, integration being a non-issue, and NUnit the easier to use, the verdict is clear: use NUnit. It's available through NuGet when you need to add it into your solution, and you can learn more at NUnit.org. Enjoy!

EDIT (6/18/2015):
The syntactical differences section omitted an item that is now included.

An evaluation of Silverlight and XAML

Up till now, I've only used this blog to post about technical issues and development patterns, without really editorializing. This entry, however, is going to be a rather subjective evaluation of technology stacks.

One of the bigger paradigm shifts in .NET development that has occurred in the last few years is the introduction of Silverlight. I'm not going to go into the reasons that Silverlight was introduced or why it has seen a significant rate of adoption (partly because the reasons for the latter are incredibly varied depending on company and application). Hand in hand with Silverlight has come XAML. The two are not one and the same: Silverlight is an application framework that's very tightly coupled to Internet solutions, while XAML is a markup language that can be used in many different types of .NET projects. This post will discuss the pros and cons of both.

Silverlight is nice in that it further simplifies web application programming. Some websites are actually not suited to a model where the user can control page flow, and some user interfaces can only be accomplished in an HTML+JavaScript environment after much trickery and tweaking third-party controls (e.g. jQuery UI). Silverlight allows you to deliver an application in a sever-client and/or web-based manner without having to dance around the stateless, static-content-oriented HTTP protocol.

Silverlight's architecture has some good ideas behind it. Applets have been a source of (real and perceived) security concerns for a long time, so the Silverlight designers decided that Silverlight would be a subset, instead of a super-set, of the .NET framework. In other words, Silverlight uses a smaller selection of .NET's functionality. This deliberate scope restriction takes away the ability of Silverlight applets to do some dangerous things. The code to access local file systems, databases, and other critical resources is not even there. It enforces a safer application ecosystem by design instead of by potentially breakable (and in the end arbitrary) access switches. It also solves (or at least mitigates) another concern of applets. Applets have to have their own execution sandbox that the client has to download, in addition to downloading the applet itself. A smaller functionality set compiles to smaller binaries, which results in smaller download and install footprints. Even with today's fast connections, dual-core processors and hundred-gig hard drives, all resources are still finite, and so creating a paradigm that rewards lighter-weight deliverables is a very smart idea.

These benefits do not come without some significant hassles, though.

One of the biggest problems I've had with Silverlight is that various versions are not compatible. You can install and run .NET 1.1, 2.0, and 4.0 all on the same machine without any problem - you can have solutions that have projects in all different versions; you can have multiple IIS app pools running different framework versions all running simultaneously. Not only is this possible, but it's extremely easy - the setup and execution is all seamless. .NET is certainly not the only application framework or product that can support this kind of parallelism, but the point is that it does it well. Silverlight does not. I know it is possible to get Silverlight 3 and 4 running on the same machine - I've seen co-workers get it done - but its a very difficult process, and even those co-workers throw up their hands in surrender when I ask them to help me re-create it. "I just un-installed and re-installed things in an apparently arbitrary order until it started working" was the answer I got from more than one of them.

On the surface this sounds like a nit-picky concern - "just use the same version of Silverlight for everything", right? But let's be realistic, it's never that simple. Various applications are developed under different constraints and requirements, and sometimes using only one version is simply not a realistic option. Some clients and environments require an older framework, and you can't change that. Plus, even if you do have the option to upgrade, development hours are limited and business users/clients aren't always willing to assume the upgrade risk. This is true for anything, not just Silverlight - there are still many .NET 2.0 applications and DLLs running in production environments that won't be upgraded for years to come for these very same reasons. Effective multi-version support is a feature I don't think enterprise software development tools can skimp on, and I feel that Silverlight not only skimped, but completely dropped the ball.

This versioning/parallelism flaw is major, but there are also some important minor annoyances. The subset mentality that Silverlight was designed in is a good idea that was clumsily executed. Instead of Silverlight being a 'true' subset of .NET, it is actually a parallel, minimized fork of .NET. It looks like .NET, it smells like .NET, but it doesn't taste like .NET. Visual Studio is always cranky when you try to add a reference to a Silverlight project in a non-Silverlight project - it'll do it, and the solution will compile, but VS will always mark it as a broken reference in Solution Explorer. Tools like ReSharper will even give you pre-compile errors in non-Silverlight code that references Silverlight code (as well as in the solution-wide analysis, which is much harder to ignore).

The path of work-arounds this particular flaw sent me down was a real comedy of errors. "Hmm, VS2010 + Resharper doesn't play nice with MSTest projects referencing Silverlight projects. Okay, let's create a Silverlight test project. Hmm, no such project type. Okay, we'll just create a Silverlight class library - the whole 'test project' definition is a somewhat arbitrary distinction anyway. Argh, okay, where can I download the MSTest for Silverlight framework? Geez that's hard to find. Okay got it! Compiles, woo-hoo! What? The MS Test runner can't run the MSTest for Silverlight tests (doesn't recognize the attributes)? Crap. Well, maybe there's a port of the test runner for Silverlight. Uh ... well there's a crappy browser-based version that's difficult to use and hard to see test failures in... Okay, screw it, let's just go with NUnit, I've never really liked MSTest anyway. NUnit port for Silverlight? Unofficially done, but existent and stable! Score one for open-source! Create Silverlight class library, add tests, reference NUnit for Silverlight DLLs ... compiles! Woo-hoo! RUNS! Woo-hoo!"

I share that bit partly to inject a little levity, partly to show that with Silverlight NUnit is nicer than MSTest, and partly to reinforce my argument that Silverlight is a second-class citizen even in the Microsoft world. MSTest doesn't like it, Visual Studio doesn't like it, it doesn't even like itself. There are just so many little 'gotchas' in trying to use Silverlight, functionality that has to be re-created, or specialized ports of existing tools that you have to employ. The whole paradigm just seems to work against code re-use, which is something that makes me rather cranky.

XAML is the markup language that Silverlight uses to create its user-interface components. As mentioned previously, though, it is not tied to Silveright. The Windows Presentation Foundation (WPF), which is intended for desktop applications, also uses XAML. In fact, Visual Studio 2010 itself is written in WPF, and therefore XAML. XAML is a big leap forward in terms of simplicity and portability of UI design - it takes everything that was great about HTML, CSS, and Web Forms, and combines them all into something even better. It further closes the gap between Windows Forms and Web Forms - these two technologies used extremely similar but inherently separate structures, but now everything is united under one roof. You can design for the desktop or the web (as long as that web is Silverlight) using one approach. XAML makes formatting pages/screens much, much more intuitive than setting up CSS stylesheets or creating application themes, and it makes the flexibility of HTML layouts available to desktop apps. Making a desktop application look pretty is no small feat regardless of technology, and WPF gives you a shorter path.

Unfortunately, XAML also takes everything that was bad about ASP.NET Data Grids and makes it the standard. The Model-View Model pattern that XAML is intended to employ encourages injecting property, method, and even class names directly in the XAML markup, or in other words, into uncompiled text. I shudder every time I see this kind of thing being done, whether it's in .config documents, vanilla XML, or in 'magic' strings inside the C#/VB code. Doing this kind of thing works against refactoring. As far as I'm aware, there exists no tool that will extend object refactors into the XAML. Given the fluid nature of the XAML data-binding model, it's a difficult task to hope to accomplish, especially considering the fact that the source object doesn't have to be bound in until run-time. Again, this may seem nit-picky, but I argue that it is not. Code is always changing, and needs to be flexible enough to accommodate rapid change. This need becomes more and more pressing each year. Members in XAML {Binding} or {StaticResources} statements are disconnected from the code in a way that discourages and complicates changes. What's even more concerning to me is that it is very easy for incomplete refactors to go unnoticed. It is very easy to change something, have the {Binding} member no longer match, and then that element no longer shows up on the screen, and no one would even notice, even with the greatest QA department in the world, because no error is thrown when said binding fails. This kind of thing has bitten us more than a few times even with the more strict binding mechanism of ASP.NET Data Grids, sometimes even in production code. I am pessimistic that such occurrences will only increase in a world that relies more heavily on XAML-based implementations.

Now, the good news is that there are ways to get around this flaw in XAML. The traditional, explicit data-binding model of giving controls names and wiring them up in the code-behind can be employed. There are also code-only ways to create {Binding}s using only C# code (no XAML) - they're not as pretty, but they work, and because they eschew magic-string based reflection they are refactor-friendly. I hope to post some examples of my own here before too long.

My current evaluation of Silverlight is that it has too many flaws to justify the somewhat dubious benefits it brings. In the end, traditional ASP.NET websites with a liberal amount of jQuery can provide all the same functionality without any of Silverlight's limitations or contrivances. And if the HTML 5 standard can ever see wide-spread adoption, then Silverlight becomes even less attractive. I would urge .NET developers to discourage the use of Silverlight in order to shorten the time till its end-of-life date.

My current evaluation of XAML (and I reserve the right to modify this in the future) is that it is better than both Windows Forms and Web Forms. While it has some non-trivial pitfalls, they are worth the risk for the benefits gained. I would urge the use of WPF for desktop development, and if it ever becomes available for non-Silverlight ASP.NET use, then it is preferable to 'pure' HTML+CSS.

Comments, questions, and corrections are more than welcome!

C# Compiler Bug, or just something obscure and frustrating?

Earlier today, I started getting the strangest error message when trying to run my tests.  The message varied a little depending on what test runner was being used, but the gist of the error message was:

"Could not load file or assembly or one of its dependencies.  Signature missing argument. (Exception from HRESULT: 0x801312E3)"

The Visual Studio solution this started occurring in is pretty simple.  It contains four (4) C# 4.0 class libraries, and two .NET 4.0 test projects.  Both test projects are using NUnit 2.5.7 and Rhino Mocks 3.6. One test project was working just fine, but when I tried to run the second test project's tests, I would get this error.  It occurred no matter if I ran it through ReSharper or through the NUnit GUI.

After three hours of trial and error, I finally found that the error appeared to be with a reference to the project containing the domain objects.  To better illustrate, here is the project hierarchy:

Test Project
    Class Library A
        Domain Project
        Class Library B
            Domain Project
        Class Library C
    Class Library B
        Domain Project
    Class Library C

It's not the most straightforward tree, but not complex by any means, and compiles without error or warning.  Yet for some reason the test runners were very upset that Class Library B was referencing the Domain Project.  It appeared to be at least in some way related to Rhino Mocks - when I removed the lines of code in the Test Project that included calls to the Expect() method, but left the actual project hierarchy the same, the error went away.  (The tests of course were then useless, so this wasn't a viable alternative, but it got me closer to finding the problem.)

The tests themselves are setting up expectations on a method from Class Project B that has a class from Domain Project as its return type.  You may notice, however, that Test Project does not reference Domain Project.  Because this solution is relatively new, so far the tests just verify that the method is called with the right inputs; I haven't yet written the tests to verify output.  In other words, I don't have to deal directly with the class from Domain Project yet, so I haven't referenced that project.

It eventually turned out that this was in fact the problem.  When I added a reference to Domain Project to Test Project, the errors went away and I was able to run my tests again. It seems that Rhino Mocks requires direct references to all the types employed by a method signature, even if the C# compiler doesn't need them all to build the DLL.  It makes it so the error ends up in a bizarre no-man's land - it's not a compile-time problem, but it manifests when the DLL is being loaded, which is before what we typically think of being run-time.

I'm not one who understands compiler design and implementation very well, so it's hard for me to say what the compiler's doing here. From comparing disassembles of the DLL compiled with and without the Domain Project reference, though, it's pretty clear that without the project reference, the compiler doesn't know the return type of delegate passed into Expect(), and can't build the method signatures correctly.  (See the footnotes for more detail.)

Honestly, this feels like something the compiler should at least send up a warning about, or perhaps even fail to build on. It results in a compiled DLL that can't be used; it feels like it allows us to create invalid binaries.  Maybe detecting this kind of problem would be so enormously complex that its better to put the burden on the developer, but you'd think they'd have better documented it in that case.

So the final take-away is: when using generics and/or delegation, make sure all types implicitly referenced by your code are explicitly referenced in the project References.

This is a very remote and unusual case, but I could find absolutely nothing on Google or in any Microsoft documentation that gave any hints, and the error message itself was basically useless.  So I am putting this recap out on the Internet in the hopes that if anyone else ever runs into this, they'll have a little more insight than I did.

Footnotes

This is the C# code written:

[Test]
public void MyTest()
{
  _classFromProjectB
    .Expect(x => x.GetBatch(Arg<int>.Is.Anything, Arg<DateTime>.Is.Anything));

  // invoke the method being tested
}

Without the Domain Project reference, this is what the compiler produces:

[CompilerGenerated]
private static byte CS$<>9__CachedAnonymousMethodDelegate1;

[CompilerGenerated]
private static IClassFromProjectB <MyTest>b__0(void x)
{
  byte CS$1$0000 =
    (byte)x.GetBatch(Arg<int>.Is.Anything, Arg<DateTime>.Is.Anything);
  return (IClassFromProjectB) CS$1$0000;
}

[Test]
public void MyTest()
{
  if (CS$<>9__CachedAnonymousMethodDelegate1 == 0)
  {
    CS$<>9__CachedAnonymousMethodDelegate1 =
      (byte) new int(null, (IntPtr) <MyTest>b__0);
  }
  this._classFromProjectB.Expect<IClassFromProjectB, byte>(
     (Function<IClassFromProjectB, byte>) CS$<>9__CachedAnonymousMethodDelegate1);

  // invoke the method being tested
}

With the project reference, it produces:

[Test]
public void MyTest()
{
  this._classFromProjectB
    .Expect<IClassFromProjectB, List<DomainObject>>(
      delegate (IClassFromProjectB x) 
      {
        return x.GetBatch(Arg<int>.Is.Anything, Arg<DateTime>.Is.Anything);
      });

  // invoke the method being tested
}