Banner reads: Testing in Real Life Part 1

How to Know What to Test: Testing IRL Part 1

Reading Time: 6 minutes


This is part one in a mini-series – Testing in Real Life – that aims to share practical information about testing, based on real life examples.  It’s born from the observation that a lot of resources I’ve seen appear to concentrate mostly on theory, or how to advance existing experience, but there’s not a lot about how to actually do these things in real life, or get started with them.  The information I’ll share in this mini-series is based on my recent experience of helping someone test their mobile app for the first time, as an example of how to test in real life.


Part one is a follow-on to my answer to “how do you know what to test?” on Ministry of Testing’s Club.



1. Get Briefed


When tasked with testing something for the first time, remember that testing isn’t just about getting a product in your hands and seeing what it does.  When we expand outwards with our testing (more commonly referred to as “shift left” and “shift right”), we understand that testing is not a phase or single activity.  We can start testing before the product even exists.


With this in mind, the first thing I did when testing the application was to ask for a briefing with the developer, who also happened to be the product owner, CEO, co-founder – basically every technical role in the two-person operation.


Here’s some of what we covered in the briefing:

  • The product’s purpose, mission and goal
  • The current state of affairs (e.g., current stage in the product lifecycle; finished features; known issues; most important features or areas to cover; low priority topics)
  • How I could add value (e.g., what I was actually being asked to test; what kind of information the team wanted to uncover; questions that testing could help answer)
  • Getting access to the database and admin panel as well as the app itself
  • Key contacts that I could consult with


Since I’d be the first person to ever test this app, I also asked about communication preferences:

  • How should I provide feedback
  • What level of detail is prefered
  • How should we track issues?

I’ll talk about those points more in part four of the mini-series.


Can’t get anyone to brief you?  Brief yourself!  The app I tested only had two people behind it and there was no documentation, but if you’re hired to test a product, there will usually be at least something that you can refer to on your own.  Like I said in my original Club answer, this could include:

  • Official product documentation
  • User stories and bug tickets
  • Public marketing information
  • Support pages

All of which are full of ideas for what to test.


If there’s really nothing available at all to help you learn about the product without directly using it or speaking to someone, this might indicate another problem (bug!) that’s worth highlighting.  Take some initiative and don’t just complain about the lack of information. Come to the discussion with a solution to propose, that would have helped you.



2. Experience It for Yourself


Once you have some idea of what the product is and what you’re actually being asked to do (the “how can I add value” part of the briefing, which you can also deduce yourself in many cases), it’s time to get your hands on the product and experience it for yourself.


If you’re testing a specific user story or bug ticket then this will largely guide the target and focus of your testing.  But what if you have a completely blank canvas, free reign, or (as can often be the case when starting on a new project) are simply told: “have a look at the app and see what you can find”?


I had complete free reign, so I started out with a paddling pool tour, which is a testing tour that I created the first time I tried session-based test management.


I explained it quite nicely in that post (if I do say so myself):


“As it sounds, the Paddling Pool Tour is a shallow dabble in test the waters, without getting too deep or detailed.  This would also allow me to cover a number of ideas in a short space of time, without having to guess in advance which one(s) might merit more testing time.  I could use the information I gathered to influence future test sessions.”


I really like doing paddling pool tours on software that’s new to me because it’s a very quick and effective way to:

  • Gather first impressions of the product as a whole
  • Get a sense of how intuitive the product is, and how clear or unclear its purpose and functions are
  • See what is already grabbing my attention and making me want to explore
  • Gather information to decide how to slice the product, topics and risks (e.g., features, UX, data, security, empty spaces, error states, branding)
  • Find inspiration for test ideas and allocate them to the appropriate slices
  • Divide these different ideas, areas, topics, etc. into manageable test sessions with test charters


A paddling pool tour is just an introduction, so don’t get too intimate with the product straight away.  Take your time and use the information you get from the paddling pool tour to generate ideas and help you form a solid test strategy.  Remember as well that you’ll only ever get this first introduction once.  You don’t yet have the curse of knowledge, so can share your thoughts and insights about your first impressions with the team, so they can understand what a user’s first impressions might also be like.

  • What surprised you?
  • What was missing?
  • What did you like?
  • Did your impression of the product match how it’s marketed?



3. Explore Different Uses, Risks, and Themes


Armed with information from first and second hand sources, you can now start to get a bit deeper into testing the product.  I had lots and lots of ideas and test charters after completing a paddling pool tour of the application.  I documented them all to make sure I wouldn’t forget.


My next step was to perform test sessions with a few of the test charters that focussed on the features and use cases that the team told me were the main purposes of having the app, and that they thought would be the most used.


The decision to start with these areas was a mixture of business priority, probability, coverage, and risk.  By testing targets that are expected to be highly used and highly profitable sooner rather than later, you increase your chances of finding high value issues sooner as well, thus adding value sooner.  If you ever feel like you don’t have enough time to test everything you want to, use this strategy to prioritise the order of what to test.


As I explored the key features and use cases, I paid attention to:

  • Things that caught my eye straight away (i.e., things that users might immediately like or dislike, which could influence whether or not they’d continue to use the product)
  • Things I noticed along the way that distracted me or made me wonder (e.g., a behaviour that was weird the first time, but became more and more bothersome; a strange behaviour or element that I didn’t understand the purpose of)
  • How actions or data in one feature or area might interact with or affect another feature or area


Even the simplest of software can have many intertwining paths.  Always keep your eye out for the potential to have crossed wires and broken bridges – otherwise known as dependencies!


The prioritising approach I described above can also be applied to the different themes and risks gathered thus far (from the briefing, the paddling pool tour, etc.).  There are many other methods to gather ideas and prioritise, but in this post, I’m just focusing on what I actually did in this case.



4. Use Retests as an Opportunity


When it comes to testing bug fixes, it can be very tempting to just check for the exact bug, the exact way it was originally found, and then move on if it seems to work as desired.  Don’t do that!  Use retests (including regression) as an opportunity to uncover information that you might have missed the first time, or even new bugs that the “fix” introduced.


When retesting, think about:

  • How the function might still be broken (e.g., other points of entry, types of data, time, location)
  • What other interesting possibilities or combinations there might be (e.g., what happens if I try to do x at the same time; what happens if I do it in this order instead?)
  • How the issue might be more complicated than it seems (i.e., approach with suspicion)
  • Whether the change introduces any risks or undermines existing safeguards
  • Anything you (or someone else) might have forgotten to consider the first time


Also ask questions about the fix, refactoring, or whatever else you might be testing:

  • How was it implemented
  • Does the change or feature have anything to do with x
  • Is the code shared by any other feature?



5. Utilise the Data


Utilising data about a product is an important aspect that many people forget about, and it’s certainly something I could do more of too.  There are lots of ideas for what to test that you can find from:

  • Issues that users report (officially and unofficially)
  • Product reviews
  • Error logs
  • Monitoring and usage statistics


Since the app I tested in this case hadn’t yet been released to customers and didn’t have any logging or monitoring in place, this kind of data wasn’t available to me, but it would be remiss to omit this as a source of ideas.  It’s also a nice way to “full circle” the reference to expand outwards testing in point one, as using monitoring and logging to aid testing is often seen as part of “shift right” or testing in production (which doesn’t mean you stop testing pre-production, by the way).



Remember the Purpose of Testing


I hope this information is useful to you and helps you to know what to test on your own project.  Above all, always remember that the purpose of testing is to uncover information.  Use that goal to guide your testing efforts.


Once you’ve uncovered information, it’s important that you also document and share it with others, so it can be used to improve the quality of the product.


Part two of Testing in Real Life will be my third post on session-based test managementRead part two now.

2 thoughts to “How to Know What to Test: Testing IRL Part 1”

Leave a Reply

Your email address will not be published. Required fields are marked *