Why Less is More in User Experience Testing

What is User Experience Testing? 

User experience testing, or usability testing, is a vital part of the UX design process. It involves observing participants using your new app, website or other interface to carry out specific tasks. During user testing, a moderator gathers information on how users interact with the app or website and what they like and dislike.

It’s usually wise to conduct user testing at regular intervals during the design process, from the early design phase up to the launch date. 

Why is User Testing Important? 

If your website or app is difficult to use or slow to load, your target users will probably approach one of your competitors instead. 

By conducting user testing, you can:

• Highlight design problems 
• Identify opportunities for improvement
• Learn more about your target user. This includes their preferences and behaviours when using your app or website.

How Many Users Should I Test?

At first glance, user testing can seem expensive and complicated. However, even with a limited budget or tight deadline, it is possible to gather helpful feedback from user testing. 

Jakob Nielsen and Tom Landauer researched how many users are needed for an effective usability study. As a result, they devised the following formula:

N (1- (1- L ) n )

• N represents the total number of user problems in the design
• L represents the proportion of user problems identified when testing a single user

Nielsen and Landauer noted that the typical value for L was 31%. When there were 0 test users, obviously, no usability problems were identified. However, once data was collected from a single test user, useful insight was available straight away. In fact, they found that collecting data from one user means you gather around 31% of the available information on the usability of your design. 

So what happens when you test a second user? Well, you’ll probably discover that the second user performs some of the actions in the same way as the first user. That means you can expect to see some overlapping data. But there’s no such thing as the ‘average user’ – everyone works differently, so you’re likely to notice some new themes too. And while the second user will usually provide some new insight, it’s unlikely to be quite as much as the first user. 

It’s the same story with the third user. They will probably do many of the same things you observed with the first and second users. You might notice that all three users perform certain actions in the same way. Again, the third user will offer new insight, but it will be less than the first and second users. 

And so it goes on. By adding more users, you actually learn less – because you end up observing the same things over and over again. There is little value to observing the same things multiple times. Once you’ve tested five users, your time is better spent tackling the usability issues that have been highlighted – not testing more users. 

Spending too much time or money on usability testing is usually a waste of resources. According to Nielsen, the best results come from testing with five users but conducting as many small-scale user tests as possible. 

The curved line flattens at around 15 users on Nielsen and Landauer’s chart. This shows that if you want to identify every single usability problem, you will need to test at least 15 users. 

However, even if you have the funds available to test 15 target users, it would be wiser to use your budget to run several small-scale tests. You might consider running three separate studies at different stages of the design process, with five users allocated to each. 

After running user testing with five participants, you can expect to identify around 85% of usability issues. You can use this data to make the required changes to the design, before running further usability tests with five new participants. 

Remember, your objective for user testing is to make improvements to your design – not highlight its weaknesses. 

The Power of Five

Even if you’ve taken the feedback from your first five user testers on board, and fixed the issues raised, the user testing process shouldn’t end there. By running a second phase of user testing, you can observe five different users to find out whether the changes you’ve made have improved usability. Be careful – in some cases, you might have inadvertently introduced a new usability issue. 

The second phase will help to highlight any issues that were missed during the first phase. It also allows you to dig deeper for issues around task flow, information architecture, and whether your design meets the needs of your target audience. 

After the second study, you can expect to have a new, shorter list of problems to address. You can tackle these in the redesign before carrying out a third and final phase of user testing. 

The bottom line? Running three separate studies with five users each can provide better insights on user experience than a single study with 15 users.

Why Can’t I Just Test With a Single User?

If less is more, you might think it’s better to test a single user. And while it would probably be quicker and cheaper, it’s not a great idea. The results are likely to be misleading, especially if the user carries out certain actions by mistake. Also, the data you gather won’t provide enough insight into areas like diversity in user behaviour. 

However many users you decide to test, there is always a fixed initial outlay to plan, set up and run your study. Choosing to test between three and five users is a good way to spread the start-up costs across your findings. 

Are There Any Exceptions? 

As with anything, there are always exceptions. Sometimes you will need to test more than five users. 

The most obvious example is when your website, app or product attracts several distinct user groups. For example, this could be a healthcare system used by both healthcare staff and service users. In this scenario, you would need to test the design with people from both groups. The same would apply to education websites used by teaching staff and students. 

If your website is targeted at several distinct user groups, you will probably still identify some themes in user testing. Many usability problems are related to how people use the internet. They can also be linked to external influences on user behaviour from other websites and systems. 

When testing two groups of users, Nielsen recommends testing three or four users from each category. When testing three or more groups of users, try running tests with three users from each category. 

Summary

When it comes to user testing, five is the magic number. If you want to keep your costs down and your user testing process simple, you should aim to observe five users during each phase of the user experience testing process. This will allow you to identify most of the key usability problems, and make the required changes to ensure a good user experience.