NSubstitute and the search for perfect error messages

Those of us that practice TDD daily already know how important good error messages in tests are. After all writing a failing test that clearly states what functionality the program is missing is the first step in TDD cycle. The rest of us that either can’t or simply don’t want to practice TDD must put extra effort to ensure that tests always fail with meaningful error messages. Unfortunately, according to what I have learned from my personal experience, the most devs either don’t have enough time or simply don’t bother to check if their tests fail with something meaningful. For average Joe developer writing tests and making them green is already a lot of work. Things like good test names and proper error messages are often forgotten.

But the developers are not the only one here to blame. Too often tools and libraries that supposedly should make unit testing simpler and easier, generate horrible and often cryptic error messages.

In this post we will take a close look at NSubstitute, a modern and popular mocking libary for .NET and see how we can improve messages generated by its argument matchers.

Let’s start by looking at a simple test. It demonstrates how NSubstitute is often used to assert that a method was called with an argument in a certain state:

public class PlainArgument {
    public int Id { get; }
    public string FirstName { get; }
    public string LastName { get; }
    public string EmailAddress { get; }

    public PlainArgument(int id, string firstName, string lastName, string emailAddress) {
        Id = id;
        FirstName = firstName;
        LastName = lastName;
        EmailAddress = emailAddress;
    }
}

public interface IFooService {
    void DoStuff(object argument);
}

[Fact]
public void Checking_argument_using_Arg_Is() {
    // Act
    _component.DoStuff();

    // Assert
    _fooService.Received()
        .DoStuff(Arg.Is<PlainArgument>(
            e => e.Id == 9 &&
                 e.FirstName == "jan" &&
                 e.LastName == "kowalski" &&
                 e.EmailAddress == "jan.kowalski@gmail.com"
                 ));
}

When the argument passed to the checked method was in an unexpected state (e.g. first name was not “jan” but “john”), we get an error message similar to (formatting added):

Expected to receive a call matching:
    DoStuff(e => ((((e.Id == 9) AndAlso 
        (e.FirstName == "jan")) AndAlso 
        (e.LastName == "kowalski")) AndAlso 
        (e.EmailAddress == "jan.kowalski@gmail.com")))
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments indicated 
with '*' characters):
    DoStuff(*PlainArgument*)

This error message is terrible. It contains a lot of informations that are easily obtainable by looking at the test method’s source code. Yet it does not contain the most important piece of information that we need: which properties have unexpected values and what these values are.

We can slightly improve this error message by overloading ToString method on PlainArgument class. Let’s call this new class StringableArgument:

public class StringableArgument {
    // the same code as in PlainArgument
    public override string ToString()
        => $"{nameof(StringableArgument)}(id: {Id}, firstName: \"{FirstName}\", " +
            $"lastName: \"{LastName}\", emailAddres: \"{EmailAddress}\")";
}

// in a test method:
_fooService.Received()
    .DoStuff(Arg.Is<StringableArgument>(
        e => e.Id == 9 &&
             e.FirstName == "jan" &&
             e.LastName == "kowalski" &&
             e.EmailAddress == "jan.kowalski@gmail.com"
             ));

Now the error message looks similar to (formatting added):

Expected to receive a call matching:
    DoStuff(e => ((((e.Id == 9) AndAlso 
        (e.FirstName == "jan")) AndAlso 
        (e.LastName == "kowalski")) AndAlso 
        (e.EmailAddress == "jan.kowalski@gmail.com")))
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments indicated 
with '*' characters):
    DoStuff(*StringableArgument(
        id: 7, firstName: "john", 
        lastName: "doe", 
        emailAddres: "john.doe@gmail.com")*)

This is better than before. Now we can see both expected and actual values of the matched argument’s properties.

One drawback of this approach is that the quality of the error message depends on the quality of ToString implementation. If we are using AOP solution like Fody to generate ToString implementations for most of our classes, then this solution may be good enough. On the other hand if we are generating and updating our ToString methods manually (even if this means pressing a shortcut in our IDE) then I would prefer to look for a better solution that is totally automatic.

There is also another problem that we were ignoring so far. Consider what will happen if we add a new field to our StringableArgument class. Because we are using property access syntax inside of a lambda expression, our existing matchers will not only compile without any problems when we add a new field, they will also pass! In order to ensure that our matchers and tests remain valid, we must go through all argument matchers for StringableArgument class and make sure that they use the newly added field.

The above problem may be solved by moving equality checking to the StringableArgument class itself. Let’s call this new class EquotableArgument:

public class EquotableArgument : IEquatable<EquotableArgument> {
    public int Id { get; }
    public string FirstName { get; }
    public string LastName { get; }
    public string EmailAddress { get; }

    public EquotableArgument(int id, string firstName, string lastName, string emailAddress) {
        Id = id;
        FirstName = firstName;
        LastName = lastName;
        EmailAddress = emailAddress;
    }

    public override string ToString()
        => $"{nameof(StringableArgument)}(id: {Id}, firstName: \"{FirstName}\", " +
            $"lastName: \"{LastName}\", emailAddres: \"{EmailAddress}\")";

    public bool Equals(EquotableArgument other) {
        if (other is null) return false;

        return ToTuple(this) == ToTuple(other);
    }

    public override bool Equals(object obj) {
        if (obj is EquotableArgument other) {
            return Equals(other);
        }

        return false;
    }

    public override int GetHashCode()
        => ToTuple(this).GetHashCode();

    private static (int, string, string, string) ToTuple(EquotableArgument arg) {
        return (arg.Id, arg.FirstName, arg.LastName, arg.EmailAddress);
    }
}

// in a test method:
_fooService.Received()
    // NOTICE: We no longer use a lambda expression.
    .DoStuff(Arg.Is(new EquotableArgument(
        id: 9, 
        firstName: "jan", 
        lastName: "kowalski",
        emailAddress: "jan.kowalski@gmail.com")));

With this solution it is impossible to forget to update our matchers when we add a new field. We also get a slightly better error message (formatting added):

Expected to receive a call matching:
    DoStuff(StringableArgument(
        id: 9, firstName: "jan", 
        lastName: "kowalski", 
        emailAddres: "jan.kowalski@gmail.com"))
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments 
indicated with '*' characters):
    DoStuff(*StringableArgument(
        id: 7, firstName: "john", 
        lastName: "doe", 
        emailAddres: "john.doe@gmail.com")*)

So far so good. But what if our argument has ten or more properties. With complex arguments looking for a one property with unexpected value may quickly change into “Where’s Wally?” game. The only way to further improve error messages is to stop relaying on NSubstitute/hand-carfted Equals implementation and instead to use specialized assertion library like FluentAssertions or NFluent.

Here is how our test would look like if we decide to use FluentAssertions:

[Fact]
public void Catching_argument_and_checking_manually_with_fluent_assertions() {
    // Arrange
    PlainArgument arg = null;

    _fooService
        .DoStuff(Arg.Do<PlainArgument>(x => arg = x));

    // Act
    _component.DoStuff();

    // Assert
    _fooService.Received()
        .DoStuff(Arg.Any<PlainArgument>());

    arg.Should()
        .BeEquivalentTo(new PlainArgument(
            id: 11, 
            firstName: "jan", 
            lastName: "kowlaski", 
            emailAddress: "jan.kowalski@gmail.com"));        
}

The error message is:

Expected member Id to be 11, but found 7.
Expected member FirstName to be "jan" with a length of 3, but "john" has a length of 4, differs near "ohn" (index 1).
Expected member LastName to be "kowlaski" with a length of 8, but "doe" has a length of 3, differs near "doe" (index 0).
Expected member EmailAddress to be 
"jan.kowalski@gmail.com" with a length of 22, but 
"john.doe@gmail.com" has a length of 18, differs near "ohn" (index 1).

With configuration:
// (skipped)
// Here FluentAssertions describes configuration
// that it used to compare the two objects.

Not bad, I must say. We get a list of only these properties that have unexpected values. Certain messages seem a little bit too verbose for me e.g. Expected member FirstName to be "jan" with a length of 3, but "john" has a length of 4, differs near "ohn" (index 1). Maybe Expected FirstName to be "jan" but was "john". would be just enough? Still it is the best solution that we have so far.

The only downside that I see is that the test code is now a little more verbose and less readable. Mainly because we are now responsible for manually capturing argument’s value:

PlainArgument arg = null;
_fooService
    .DoStuff(Arg.Do<PlainArgument>(x => arg = x));

With a bit of C# magic we may make argument capturing less painful:

_fooService
    .DoStuff(Capture(out Arg<PlainArgument> arg));

// Act
_component.DoStuff();

// Assert
_fooService.Received()
    .DoStuff(Arg.Any<PlainArgument>());

// This time we use NFluent
Check.That(arg.Value).HasFieldsWithSameValues(
    new PlainArgument(
        id: 7, 
        firstName: "john", 
        lastName: "doe", 
        emailAddress: "john.doe@gmail.com"));

To see how it works please check ArgCapture.cs file.

Catching argument’s value manually is cumbersome and makes tests less readable. On the other hand using some “magical” syntactic sugar also does not looks like a good idea. After all our code should be simple. If we can avoid using “magic” we should do it!

Our final solution is to create a custom NSubstitute argument matcher. The matcher uses undercover FluentAssertions library to perform the check. Here is how the test code looks like with this approach:

[Fact]
public void Checking_argument_using_custom_NSubstitute_matcher() {
    // Arrange

    // Act
    _component.DoStuff();

    // Assert
    var expected = new PlainArgument(
        id: 11, 
        firstName: "jan", 
        lastName: "kowlaski", 
        emailAddress: "jan.kowalski@gmail.com");

    _fooService.Received()
        .DoStuff(WithArg.EquivalentTo(expected));
}

The error message generated for an argument that does not overload ToString looks like this (formatting added):

Expected to receive a call matching:
    DoStuff(PlainArgument)
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments indicated 
with '*' characters):
    DoStuff(*PlainArgument*)
        arg[0]: Expected member Id to be 11, but found 7.
                Expected member FirstName to be "jan" with a length of 3, 
                but "john" has a length of 4, differs near "ohn" (index 1).
                Expected member LastName to be "kowlaski" with a length of 8, 
                but "doe" has a length of 3, differs near "doe" (index 0).
                Expected member EmailAddress to be 
                "jan.kowalski@gmail.com" with a length of 22, but 
                "john.doe@gmail.com" has a length of 18, differs near "ohn" 
                (index 1).

It is clear that the problem occurred at the first argument (arg[0]). Also we can see the actual and expected values of the argument’s fields and properties. And the test code is simple and clean. If you are interested how it is implemented please see CustomMatcher.cs file.

As we can see there exists no perfect solution. Still with a little effort we can make our error messages much more readable and pleasurable to work with. I personally suggest to use either the last solution or the solution presented in Catching_argument_and_checking_manually_with_fluent_assertions test.

Source code and examples: GitHub

Menu