This post is inspired by several conversations I’ve had over the years, as well as my own musings about testing targets, testing intentions, and terminology. I’ve noticed that there’s often confusion and a great effort to give different types of tests (or those perceived as such) different names, and having only one name or category for each specific test. My view on this, as with many terminology-focussed topics, is that the shared understanding and intentions are the most important things.
With that being said, I hope to provide a little bit of clarity on one set of meanings and intentions when it comes to “unit tests”, “integration tests”, “regression tests”, and some other terms often used in the same conversations.
Unit Tests
There’s a lot of debate around what is mean by a “unit test”. Probably the most common understanding is that a unit test is a test of one specific function or method, and nothing more. For this reason, unit tests are often very closely coupled with implementation methods, and I’ve heard this being raised as a reason why test driven development (TDD) is difficult or impractical for some, as they don’t yet know which functions they’ll write.
I have a more flexible view, which includes another understanding. In my view, a “unit” is something small, yet meaningful. What is more meaningful to me than an individual function or method is a unit of business functionality or value. Some behaviour or process where, if something were to go wrong, this would be recognised as a bug.
You may argue that failing tests of individual functions would also lead to bugs, and this is true. But a collection of unit tests on a function level might not detect a bug which a test on a business level would. Why is this important? I’ll come to that when discussing what kind(s) of tests to use.
An advantage of function-based unit tests is that you get more specific information on what went wrong. An advantage of business-based unit tests is that you don’t have to update them every time an implementation detail changes, making them especially useful when refactoring, or practising TDD. All you need to know is what you put in, and what you should get out.
Integration Tests
You may read my “business unit” interpretation of a “unit test” and say, “well, that’s an integration test,” and in some cases, that might be true. We can think of integration tests as tests which check the interaction of multiple functions or methods, which is, again, probably the most common understanding. But we can similarly think of them as checks between multiple business functionalities or behaviours.
A test can simultaneously check a single business behaviour and multiple code methods. A test can be both a “unit test” and an “integration test”, depending on your understanding of those terms.
How to Decide What Kind of Test to Use
Let’s assume that you accept my interpretations of “unit tests” and “integration tests”, and can work with them. How can you decide which kind(s) of tests to use in a given scenario? Before we can answer that question, we must consider why we even have tests in the first place.
I think this is something we often forget about, especially when we’re given code coverage percentage goals. But tests aren’t something we should create because “we have to”. We should create tests as tools to help ourselves, now and in the future.
Good tests provide feedback. Good tests detect change. Good tests are something you pay attention to.
With that, let’s think again about what we’re trying to test. We’re trying to test that our code works, right? But what does that mean? Usually, that means that it should fulfil the business’ acceptance criteria. So again, the business unit seems to be more important than the code unit. Of course, a team or project should always consider tech excellence and good system architecture, but when it comes to whether or not something can be released, we usually care more about whether it meets the business requirements than which methods or code were involved.
That’s not to say we don’t care about code units; we do. We care about where exactly the offending code is, which caused the bug. Therefore, in my view, it’s important to have both kinds of unit tests (code- and business-based), and whether they end up being “true” unit tests, in the sense of checking an individual function, or integration tests is less important.
All that to say, if the business functionality or behaviour is important, write a test which specifically covers it, based on input and output. And if you would benefit from more specific information on the origin of a bug in the code, write a test which would give you that specific information.
When deciding whether or not your tests are sufficient, ask yourself: If would these catch changes in behaviour that would be considered bugs? If there was a bug in this feature, would I know where to fix it?
Regression and Other Tests
So far, we’ve been talking about what we want to test, and we’ve discussed terms which represent those intentions. But there’s another set of terminology that gets somehow confused with this. That’s the why we want to test.
When you think of “regression tests”, your mind may jump to scripted test cases, or automation on a higher level, like via the UI. In reality, “regression” doesn’t describe what you want to test, it describes why – to check that the quality of the system under test (SUT) hasn’t gotten worse since your changes; that it hasn’t regressed.
A test can simultaneously check something small and meaningful, and check that it wasn’t altered by a code change. A test can be both a “unit test” and a “regression test”.
If you’re already writing unit and integration tests as you implement new features, you’re already creating a set of regression tests. They don’t have to be separate tests, and they don’t have to be the only tests. That is to say: Just because you don’t have higher level tests, doesn’t mean you don’t have regression tests, and just because you have unit and integration tests, doesn’t mean you can’t or shouldn’t have higher level tests too.
The ideal scenario is for tests to be written alongside – or even before (TDD) – implementation, rather than going back and creating tests for things which have long existed, although this is often also done to fill gaps and enhance coverage. While the feature is being implemented, you may call these feature tests. Once the feature is done, they become regression tests, so long as you keep executing them.
In the same way that feature tests can become regression tests, a subset of regression tests can be used as smoke tests (these are normally on higher levels) – a limited set of checks intended to check the most important or basic functionality of the SUT before more detailed testing is performed.
A test can simultaneously check for changes and give feedback on whether more detailed testing is worthwhile at a given point in time. A test can be both a “regression test” and a “smoke test”.
Should You Write a Regression Test for Every Reported Bug?
The short version is no. The longer version is that the answer is based on risk and return on investment (ROI). How likely is this bug to occur again? How catastrophic would it be if this bug were to occur again? How often would you want to execute this test? And how long would it take you to write, run, and maintain the test(s)? In the end, it’s a judgement call, based on how panicked you would feel if the bug occurred again and someone discovered that you didn’t create a test for it before.
This post obviously didn’t cover all the types of tests out there – from a “what” or “why” perspective. But hopefully it’s enough to explain what can be meant by different testing terms, and make things a little less confusing. What are your thoughts on these terms? Do you use them in other ways I didn’t mention? Share your thoughts in the comments.
Discover more from Cassandra HL
Subscribe to get the latest posts sent to your email.