Those of us that practice TDD daily already know how important good error messages in tests are. After all writing a failing test that clearly states what functionality the program is missing is the first step in TDD cycle. The rest of us that either can’t or simply don’t want to practice TDD must put extra effort to ensure that tests always fail with meaningful error messages. Unfortunately, according to what I have learned from my personal experience, the most devs either don’t have enough time or simply don’t bother to check if their tests fail with something meaningful. For average Joe developer writing tests and making them green is already a lot of work. Things like good test names and proper error messages are often forgotten.
But the developers are not the only one here to blame. Too often tools and libraries that supposedly should make unit testing simpler and easier, generate horrible and often cryptic error messages.
In this post we will take a close look at NSubstitute, a modern and popular mocking libary for .NET and see how we can improve messages generated by its argument matchers.
Let’s start by looking at a simple test. It demonstrates how NSubstitute is often used to assert that a method was called with an argument in a certain state:
public class PlainArgument {
public int Id { get; }
public string FirstName { get; }
public string LastName { get; }
public string EmailAddress { get; }
public PlainArgument(int id, string firstName, string lastName, string emailAddress) {
Id = id;
FirstName = firstName;
LastName = lastName;
EmailAddress = emailAddress;
}
}
public interface IFooService {
void DoStuff(object argument);
}
[Fact]
public void Checking_argument_using_Arg_Is() {
// Act
_component.DoStuff();
// Assert
_fooService.Received()
.DoStuff(Arg.Is<PlainArgument>(
e => e.Id == 9 &&
e.FirstName == "jan" &&
e.LastName == "kowalski" &&
e.EmailAddress == "jan.kowalski@gmail.com"
));
}
When the argument passed to the checked method was in an unexpected state (e.g. first name was not “jan” but “john”), we get an error message similar to (formatting added):
Expected to receive a call matching:
DoStuff(e => ((((e.Id == 9) AndAlso
(e.FirstName == "jan")) AndAlso
(e.LastName == "kowalski")) AndAlso
(e.EmailAddress == "jan.kowalski@gmail.com")))
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments indicated
with '*' characters):
DoStuff(*PlainArgument*)
This error message is terrible. It contains a lot of informations that are easily obtainable by looking at the test method’s source code. Yet it does not contain the most important piece of information that we need: which properties have unexpected values and what these values are.
We can slightly improve this error message by overloading ToString
method
on PlainArgument
class.
Let’s call this new class StringableArgument
:
public class StringableArgument {
// the same code as in PlainArgument
public override string ToString()
=> $"{nameof(StringableArgument)}(id: {Id}, firstName: \"{FirstName}\", " +
$"lastName: \"{LastName}\", emailAddres: \"{EmailAddress}\")";
}
// in a test method:
_fooService.Received()
.DoStuff(Arg.Is<StringableArgument>(
e => e.Id == 9 &&
e.FirstName == "jan" &&
e.LastName == "kowalski" &&
e.EmailAddress == "jan.kowalski@gmail.com"
));
Now the error message looks similar to (formatting added):
Expected to receive a call matching:
DoStuff(e => ((((e.Id == 9) AndAlso
(e.FirstName == "jan")) AndAlso
(e.LastName == "kowalski")) AndAlso
(e.EmailAddress == "jan.kowalski@gmail.com")))
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments indicated
with '*' characters):
DoStuff(*StringableArgument(
id: 7, firstName: "john",
lastName: "doe",
emailAddres: "john.doe@gmail.com")*)
This is better than before. Now we can see both expected and actual values of the matched argument’s properties.
One drawback of this approach is that
the quality of the error message depends on the quality of
ToString
implementation.
If we are using AOP solution like
Fody to generate ToString
implementations
for most of our classes, then this solution may be good enough.
On the other hand if we are generating and updating our ToString
methods
manually (even if this means pressing a shortcut in our IDE)
then I would prefer to look for a better solution that is totally automatic.
There is also another problem that we were ignoring so far.
Consider what will happen if we add a new field to our StringableArgument
class.
Because we are using property access syntax inside of a lambda expression,
our existing matchers will not only compile without any problems when
we add a new field, they will also pass!
In order to ensure that our matchers and tests remain valid,
we must go through all argument matchers
for StringableArgument
class and make sure that they use
the newly added field.
The above problem may be solved by moving equality checking
to the StringableArgument
class itself.
Let’s call this new class EquotableArgument
:
public class EquotableArgument : IEquatable<EquotableArgument> {
public int Id { get; }
public string FirstName { get; }
public string LastName { get; }
public string EmailAddress { get; }
public EquotableArgument(int id, string firstName, string lastName, string emailAddress) {
Id = id;
FirstName = firstName;
LastName = lastName;
EmailAddress = emailAddress;
}
public override string ToString()
=> $"{nameof(StringableArgument)}(id: {Id}, firstName: \"{FirstName}\", " +
$"lastName: \"{LastName}\", emailAddres: \"{EmailAddress}\")";
public bool Equals(EquotableArgument other) {
if (other is null) return false;
return ToTuple(this) == ToTuple(other);
}
public override bool Equals(object obj) {
if (obj is EquotableArgument other) {
return Equals(other);
}
return false;
}
public override int GetHashCode()
=> ToTuple(this).GetHashCode();
private static (int, string, string, string) ToTuple(EquotableArgument arg) {
return (arg.Id, arg.FirstName, arg.LastName, arg.EmailAddress);
}
}
// in a test method:
_fooService.Received()
// NOTICE: We no longer use a lambda expression.
.DoStuff(Arg.Is(new EquotableArgument(
id: 9,
firstName: "jan",
lastName: "kowalski",
emailAddress: "jan.kowalski@gmail.com")));
With this solution it is impossible to forget to update our matchers when we add a new field. We also get a slightly better error message (formatting added):
Expected to receive a call matching:
DoStuff(StringableArgument(
id: 9, firstName: "jan",
lastName: "kowalski",
emailAddres: "jan.kowalski@gmail.com"))
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments
indicated with '*' characters):
DoStuff(*StringableArgument(
id: 7, firstName: "john",
lastName: "doe",
emailAddres: "john.doe@gmail.com")*)
So far so good. But what if our argument has ten or more properties.
With complex arguments looking for a one property with
unexpected value may quickly change into “Where’s Wally?” game.
The only way to further improve error messages is to stop relaying on
NSubstitute/hand-carfted Equals
implementation
and instead to use specialized assertion library like
FluentAssertions or
NFluent.
Here is how our test would look like if we decide to use FluentAssertions:
[Fact]
public void Catching_argument_and_checking_manually_with_fluent_assertions() {
// Arrange
PlainArgument arg = null;
_fooService
.DoStuff(Arg.Do<PlainArgument>(x => arg = x));
// Act
_component.DoStuff();
// Assert
_fooService.Received()
.DoStuff(Arg.Any<PlainArgument>());
arg.Should()
.BeEquivalentTo(new PlainArgument(
id: 11,
firstName: "jan",
lastName: "kowlaski",
emailAddress: "jan.kowalski@gmail.com"));
}
The error message is:
Expected member Id to be 11, but found 7.
Expected member FirstName to be "jan" with a length of 3, but "john" has a length of 4, differs near "ohn" (index 1).
Expected member LastName to be "kowlaski" with a length of 8, but "doe" has a length of 3, differs near "doe" (index 0).
Expected member EmailAddress to be
"jan.kowalski@gmail.com" with a length of 22, but
"john.doe@gmail.com" has a length of 18, differs near "ohn" (index 1).
With configuration:
// (skipped)
// Here FluentAssertions describes configuration
// that it used to compare the two objects.
Not bad, I must say. We get a list of only these properties that have
unexpected values. Certain messages seem a little bit too verbose for me
e.g. Expected member FirstName to be "jan" with a length of 3
,
but "john" has a length of 4, differs near "ohn" (index 1).
Maybe Expected FirstName to be "jan" but was "john".
would be
just enough?
Still it is the best solution that we have so far.
The only downside that I see is that the test code is now a little more verbose and less readable. Mainly because we are now responsible for manually capturing argument’s value:
PlainArgument arg = null;
_fooService
.DoStuff(Arg.Do<PlainArgument>(x => arg = x));
With a bit of C# magic we may make argument capturing less painful:
_fooService
.DoStuff(Capture(out Arg<PlainArgument> arg));
// Act
_component.DoStuff();
// Assert
_fooService.Received()
.DoStuff(Arg.Any<PlainArgument>());
// This time we use NFluent
Check.That(arg.Value).HasFieldsWithSameValues(
new PlainArgument(
id: 7,
firstName: "john",
lastName: "doe",
emailAddress: "john.doe@gmail.com"));
To see how it works please check ArgCapture.cs file.
Catching argument’s value manually is cumbersome and makes tests less readable. On the other hand using some “magical” syntactic sugar also does not looks like a good idea. After all our code should be simple. If we can avoid using “magic” we should do it!
Our final solution is to create a custom NSubstitute argument matcher. The matcher uses undercover FluentAssertions library to perform the check. Here is how the test code looks like with this approach:
[Fact]
public void Checking_argument_using_custom_NSubstitute_matcher() {
// Arrange
// Act
_component.DoStuff();
// Assert
var expected = new PlainArgument(
id: 11,
firstName: "jan",
lastName: "kowlaski",
emailAddress: "jan.kowalski@gmail.com");
_fooService.Received()
.DoStuff(WithArg.EquivalentTo(expected));
}
The error message generated for an argument that does
not overload ToString
looks like this (formatting added):
Expected to receive a call matching:
DoStuff(PlainArgument)
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments indicated
with '*' characters):
DoStuff(*PlainArgument*)
arg[0]: Expected member Id to be 11, but found 7.
Expected member FirstName to be "jan" with a length of 3,
but "john" has a length of 4, differs near "ohn" (index 1).
Expected member LastName to be "kowlaski" with a length of 8,
but "doe" has a length of 3, differs near "doe" (index 0).
Expected member EmailAddress to be
"jan.kowalski@gmail.com" with a length of 22, but
"john.doe@gmail.com" has a length of 18, differs near "ohn"
(index 1).
It is clear that the problem occurred at the first argument (arg[0]
).
Also we can see the actual and expected values of the argument’s
fields and properties.
And the test code is simple and clean.
If you are interested how it is implemented please see
CustomMatcher.cs
file.
As we can see there exists no perfect solution. Still with a little effort
we can make our error messages much more readable and pleasurable to work with.
I personally suggest to use either the last solution or
the solution presented in Catching_argument_and_checking_manually_with_fluent_assertions
test.
Source code and examples: GitHub