While trying to construct a test strategy for a difficult project, I was recently given this advice: “Don’t test thoroughly”.
The droning siren already in the back of my mind loudened to a shrill alarm bell that it made it hard to concentrate.
Don’t test thoroughly? Why do you hate me? Don’t you know me at all? Tell me that’s a joke! – the knee-jerk remarks I did well not to verbalise.
In truth, there were valid reasons for giving this advice that I grudgingly and uncomfortably accepted after further discussion, on the basis of us agreeing pre-test that this code would not be shipped without more testing than what I was about to do; what I felt to be far from adequate.
With the same feeling one might have leaving their cat alone for three days while visiting a sick relative, I returned to my desk still feeling bad about what I’d just agreed to. Kind of necessary, although maybe not…?
Knowing deep down that a “thorough test” is not always the right test, and looking for re-assurance from fellow testers, I tweeted:
To me, "don't test thoroughly" = "half-ass your job". Please remind me why that isn't always the case? Hate feeling I'm not doing my job.
— Cassandra H. Leung (@Tweet_Cassandra) February 9, 2017
The responses I received made me feel a little better. I wouldn’t instantly be banished from the testing circle for failing to uphold the moral code!
I reflected further on, “don’t test thoroughly,” and what the “real” message behind this might be, considering suggestions from those that had responded to my appeal.
“Don’t test thoroughly” could actually be a way of saying:
- We’re ready for some happy path tests, but not exploring
- The current scope for testing is narrow, but will widen before we release
- We’re aware of outstanding bugs that aren’t to be addressed at this time
- We’re aware of outstanding bugs for which a fix is in progress
- We want you to report new issues in a different way
- We’re not ready to address any other issues yet
- Only critical bugs are being looked at
- As long as there are no errors, unexpected behaviour is okay just now
- We just need the core to work right now, and will branch out later
- We don’t want a lot of time spent on this
- We’re under time constraints
- The goal for test coverage is much lower than you’ve planned
- The areas you’re testing don’t impact the value of the product
- Your tests are too geared towards edge cases
- We need more speed than perfection
The list goes on…
Looking back on how I’ve approached testing previously, I can see how some of these messages could be relevant to the project in question, and my tendencies in general.
Although the initial conversation from which this stemmed was somewhat difficult and went against the grain, it’s been a great opportunity to consider not only the underlying messages people (including myself) are trying to get across, but also what I can do to keep improving how I test and ultimately deliver value to the business.
When I likely forget all this the moment I next hear, “don’t test thoroughly,” I’ll be able to refer back to my own list of possible translations, and hopefully ask the right questions to find out what’s really being said.
At the heart of it, testing needs to be fit for purpose. So, perhaps the next time I get a request that doesn’t seem aligned with my thoughts on testing, I could start by asking, “what’s the purpose of this testing?” I may find that by more clearly identifying and understanding shared goals, a request that was initially startling might turn out to be completely reasonable.
Have you thought of any other possible translations, whether genuine or pure comedy / sarcasm? Share them in the comments!