Understanding Different Types of Automated Tests: Unit Tests

There’s a lot of confusion when it comes to test automation. One of the reasons of confusion is that most people don’t realize there are different types of tests and that each type has different approaches and advantages/disadvantages.

When I started exploring the world of test automation, I noticed that most information out there in the form of blog posts, articles, and books focuses on unit tests. When a programmer starts exploring test automation, it’s very likely they’re only going to learn about unit tests because other types of tests are not covered that much.

Now that I’ve figured it out, I still see that most programmers I work with are stuck on the unit tests level. Instead of having to explain the same thing over and over again to each of them, I decided to write a series of posts on the topic.

So let’s start with unit tests and then move to higher level tests in the next posts.

What Is a Unit Test?

A unit test tests a single unit and most of the time those units are classes. Even though there are other types of units, let’s only focus on classes since that’s what most programmers deal with.

A common confusion I see is that programmers using tools like JUnit or PHPUnit call their tests unit tests even when they write higher level tests. Just because the tool you use to write tests has a word “unit” in it, it doesn’t mean you can only write unit tests with it.

A unit test tests a unit in isolation, even if the unit being tested interacts with other units. Basically, if you write a real unit test, it won’t fail because of a bug in a collaborating unit.

For example, if you have an Emailer class and it uses a EmailAddressValidator class to validate email addresses it needs to send emails to, a bug in EmailAddressValidator will not make unit tests of Emailer fail. They will still happily pass.

If, on the other hand, a bug in EmailAddressValidator makes Emailer tests fail, those are not unit tests.

Hopefully, that makes is clear. Let’s move on.

Unit tests can be further divided into two subtypes: leaf unit tests and interaction tests. Let’s cover each in detail.

Leaf Unit Tests

Leaf unit tests are the easiest to start with because they test a leaf unit — that is, a class that doesn’t interact with any other classes. A leaf class just does its job and returns a result.

An example of a class like that would be an email address parts extractor. That class would have a method that takes an email address, splits it by the @ character, and returns two components — username and hostname.

It might also throw an exception if there’s no @ character or if there are two or more of them.

Writing tests for a class like that is very easy. Those tests would create an instance of the class they want to test, pass it multiple email addresses, and check for result. Then there would be a test passing invalid email addresses and verifying that the method under test throws a particular type of exception.

Here’s an example:

public class EmailAddressPartsExtractorTest {
    private EmailAddressPartsExtractor extractor = new EmailAddressPartsExtractor();

    @Test
    public void extractUsername() {
        assertThat(extractor.extractUsername("elnur.blog@example.com"), is("elnur.blog"));
    }

    @Test
    public void extractHost() {
        assertThat(extractor.extractHostname("elnur.blog@example.com"), is("example.com"));
    }

    @Test(expected = InvalidEmailAddressException.class)
    public void failToExtractUsernameFromAddressWithoutAtCharacter() {
        extractor.extractUsername("elnur.example.com");
    }

    @Test(expected = InvalidEmailAddressException.class)
    public void failToExtractUsernameFromInvalidAddressWithMoreThanOneAtCharacter() {
        extractor.extractUsername("elnur@example@com");
    }

    @Test(expected = InvalidEmailAddressException.class)
    public void failToExtractHostnameFromAddressWithoutAtCharacter() {
        extractor.extractHostname("elnur.example.com");
    }

    @Test(expected = InvalidEmailAddressException.class)
    public void failToExtractHostnameFromInvalidAddressWithMoreThanOneAtCharacter() {
        extractor.extractHostname("elnur@example@com");
    }
}

As you can see, the expected argument of the @Test annotation specifies that a particular exception has to be thrown. If it doesn’t get thrown, a test fails.

Any nontrivial system has at least a few leaf classes like that. Email address parts extractor, email address validator, URI builder, password encoder, password comparator, title case converter, full name builder, template renderer, etc. Most programmers categorize them as utility classes.

This is the type of tests most programmers learn and then stop without learning anything else.

Unit Interaction Tests

Interaction tests are much more interesting.

An interaction test tests how a particular unit interacts with other units in order to get its job done. Those units it interacts with are collaborators. Some of those collaborators could be leaf classes, while others could be other collaborating classes.

Let’s say you have an Emailer class. It has a method that takes an email address, a subject, and a template name that gets rendered and used as email body.

This class could be used to send an email address validation email to a newly registered user.

Now, to do its job, this class needs to collaborate with multiple other classes.

First, it needs to validate the email address. But it’s not going to it by itself; instead, it will pass the address to an EmailAddressValidator class and that one will respond with true or false to indicate if the address is valid or not.

Then it will pass the name of the template to a TemplateRenderer class. The rendered will read the template from the file system and return it as a string to the emailer.

Then the Emailer will pass all that required information to an SmtpClient class that will do whatever needs to be done to have that email out on the Internet where a bunch of intermediate SMTP servers will pass it along until it hits the recipient’s server.

As you can see, there’s a lot going on to have an email address confirmation email sent out. But our Emailer class doesn’t do all that job itself; instead it just orchestrates a particular task, breaking it down into multiple smaller tasks, and delegating each to a class that specializes on solving that task.

Basically, that’s the single responsibility the Emailer class has — to abstract away the details of what needs to be done to send out an email by providing a higher level abstraction.

When we write unit tests for our Emailer, we only want to ensure it does its job properly. We don’t want to test that of all its collaborators do their jobs properly. All we care about is that our Emailer interacts with its collaborators properly and assume the collaborators are tested separately and hence work they way they’re supposed to.

In order to do that, we need to stub out all the collaborators. We need smart stubs that can be programmed to behave in a certain way when a particular method of them is called with particular arguments. And those stubs are called mocks.

What we do is mock out all collaborators and program them to respond in a way that’s necessary to test our emailer.

Let’s go through each collaborator.

The first collaborator is EmailAddressValidator. We need to test two branches of execution of our emailer based on the return value of the validator.

We start by writing a test that checks that our Emailer throws a particular type of exception if the validator returns false. We program the validator mock to return false, call the emailer’s send() method, and check that InvalidEmailAddressException has been thrown.

@Test(expected = InvalidEmailAddressException.class)
public void throwsExceptionOnEmailAddressValidationFailure() {
    EmailAddressValidator validator = context.mock(EmailAddressValidator.class);
    TemplateRenderer renderer = context.mock(TemplateRenderer.class);
    SmtpClient smtpClient = context.mock(SmtpClient.class);

    String emailAddress = "foo.bar";
    String subject = "Please confirm your email address";
    String templateName = "email-address-confirmation";

    context.checking(new Expectations() {{
        oneOf(validator).isValid(emailAddress);
        will(returnValue(false));

        never(renderer);
        never(smtpClient);
    }});

    Emailer emailer = new Emailer(validator, renderer, smtpClient);
    emailer.send(emailAddress, subject, templateName);
}

We also add an explicit expectation to our test that mocks of all the other collaborators are not being interacted with. We do that because we want to ensure nothing happens in the case of an invalid email address.

We now need to test interaction with TemplateRenderer. We have at least 2 branches of execution here.

The first branch is when everything is okay. In that case, the rendered string is returned back to the emailer and then the emailer passes it to the SMTP client. We’ll cover that in the final test that tests the happy path scenario.

The second branch is when the specified template is not found in the filesystem. An interesting aspect here is that TemplateRenderer will throw its own low level exception, but we want a higher level exception to be thrown from Emailer.

@Test(expected = TemplateNotFoundException.class)
public void throwsExceptionOnMissingTemplate() {
    EmailAddressValidator validator = context.mock(EmailAddressValidator.class);
    TemplateRenderer renderer = context.mock(TemplateRenderer.class);
    SmtpClient smtpClient = context.mock(SmtpClient.class);

    String emailAddress = "alan@turing.dev";
    String subject = "Please confirm your email address";
    String templateName = "missing";

    context.checking(new Expectations() {{
        oneOf(validator).isValid(emailAddress);
        will(returnValue(true));

        oneOf(renderer).render(templateName);
        will(throwException(new FileNotFoundException()));

        never(smtpClient);
    }});

    Emailer emailer = new Emailer(validator, renderer, smtpClient);
    emailer.send(emailAddress, subject, templateName);
}

The renderer throws a FileNotFoundException exception. But since our email renderer is higher level and doesn’t care where templates come from — the filesystem or a database — we don’t want that exception to propagate to Emailer clients. So Emailer catches that exception and throws its own TemplateNotFoundException exception.

To do that, we need to program the mock of TemplateRenderer to throw the FileNotFoundException exception. And that’s what we do in that particular test.

And then we add an expectation that the SmtpClient mock is not being interacted with because of this failure.

Then we add a check that calling send() on the emailer throws TemplateNotFoundException. If that check passes, all is good and we can move on.

The same approach is applied when it comes to testing interactions with the SmtpClient collaborator. But in this case, we translate SmtpRelayConnectionException to EmailDeliveryException.

@Test(expected = EmailDeliveryException.class)
public void throwsExceptionOnSmtpRelayConnectionFailure() {
    EmailAddressValidator validator = context.mock(EmailAddressValidator.class);
    TemplateRenderer renderer = context.mock(TemplateRenderer.class);
    SmtpClient smtpClient = context.mock(SmtpClient.class);

    String emailAddress = "alan@turing.dev";
    String subject = "Please confirm your email address";
    String templateName = "email-address-confirmation";
    String body = "Please confirm your email address by clicking this link: ...";

    context.checking(new Expectations() {{
        oneOf(validator).isValid(emailAddress);
        will(returnValue(true));

        oneOf(renderer).render(templateName);
        will(returnValue(body));

        oneOf(smtpClient).send(emailAddress, subject, body);
        will(throwException(new SmtpRelayConnectionException()));
    }});

    Emailer emailer = new Emailer(validator, renderer, smtpClient);
    emailer.send(emailAddress, subject, templateName);
}

After testing all kinds of problems our Emailer can run into, we implement a happy path scenario test:

@Test
public void sendSuccessfully() {
    EmailAddressValidator validator = context.mock(EmailAddressValidator.class);
    TemplateRenderer renderer = context.mock(TemplateRenderer.class);
    SmtpClient smtpClient = context.mock(SmtpClient.class);

    String emailAddress = "alan@turing.dev";
    String subject = "Please confirm your email address";
    String templateName = "email-address-confirmation";
    String body = "Please confirm your email address by clicking this link: ...";

    context.checking(new Expectations() {{
        oneOf(validator).isValid(emailAddress);
        will(returnValue(true));

        oneOf(renderer).render(templateName);
        will(returnValue(body));

        oneOf(smtpClient).send(emailAddress, subject, body);
    }});

    Emailer emailer = new Emailer(validator, renderer, smtpClient);
    emailer.send(emailAddress, subject, templateName);
}

As you can see, we don’t check for any return value. All we know is that if no exception has been thrown, all went well.

And since these are unit tests, we don’t really care how collaborators do their jobs. All we care is that Emailer does its own job by exchanging information with its collaborators and that’s it. Whether or not any of its collaborators actually work is not a concern of Emailer nor EmailerTest.

That’s basically how interaction unit tests work. As you can see, there’s much more to them compared to leaf unit tests. Interaction tests use mocks to stub out collaborators of the tested unit, while leaf unit tests don’t have collaborators and hence don’t have to stub anything.

When to Use Unit Tests

The main advantage of unit tests is that they’re fast. Very fast. It takes a few milliseconds to execute each test. We can have many thousands of them and they will execute in a matter of seconds or a few minutes.

Unit tests are the only and best option when it comes to testing leaf units. Since unit tests are so fast, we can write as many of them as we want to test any possible input combination to gain confidence that units under test do their job reliably.

The main disadvantage of unit test is how low level they are. We have to specify every collaborator a class interacts with. Then we have to specify every interaction a class might have with those collaborators, including every value they exchange with them.

A problem this low level testing leads to is how much work it is to refactor the codebase. And the more unit tests you have, the bigger the problem is. For instance, if we decide to merge two classes into one or split a class into two, we’ll have to update the related unit tests, even though the external behavior of the whole system doesn’t change a bit.

There’s also a trap we can fall into if we only write unit tests. We could have a 100% coverage of the codebase of our system and yet it’s possible the system won’t work at all because the entry point method is empty.

All in all, unit tests have their place and are very useful when done properly and in the right contexts. But there are higher level types of tests that can give us enough confidence in our system without sacrificing its refactorability.

One of those higher level types of tests — component tests — are covered in the next post of the series.

Published by

Elnur Abdurrakhimov

Elnur Abdurrakhimov is a software architect and developer with over a decade of real world experience.