Usability Benchmark

A tightly scripted usability studies are performed with several participants, using precise and predetermined measures of performance - an assessment of the usability of the product. Focused on understanding the current state of the usability.

Research Details

Research Type	Sample Size	Suggested Time
Quantitative, Behavioral, Summative	Large (20+ participants)	30 minutes

When to Use

1. To find opportunities to improve upon in future project work.
2. To understand how design and functional changes impacted the user experience over time.

Steps

1. Determine what you want to learn and how it would be best measured. Its best to focus on truly quantitative measurements, such as the metrics below, though qualitative measurements can be benchmarked as well.
2. Write the script for the tasks the participant must complete to adequately capture the measurements decided in step 1. Tasks should be goal-oriented and should not lead the user.
3. Write the post-task survey. Capture some qualitative number measurements but more importantly include and open-ended question with a fill-in response.

Common Quantitative Usability Measurements

• Completion Rates
• Task Completion Time
• Time on Task
• Task Time to Failure
• Number of Errors
• Number of Corrected Errors (aka Number of Errors Submitted)
• Learnability - track above over time / repetition with same user

Task Guidelines

1. Based on customer top tasks. You must choose task questions that are examples of top tasks. If you measure and then seek to improve the performance of tiny tasks (low demand tasks) you may be contributing to a decline in the overall customer experience.
2. Repeatable. Create task questions that you can test again in 6 to 12 months.
3. Representative and typical. Don’t make the task questions particularly difficult. Start off with reasonably basic, typical questions.
4. Universal, everyone can do it. Every one of your test participants must be able to do each task. If you’re going to be testing a mixture of technical, marketing, and sales people, don’t choose a task question that only a salesperson can do.
5. One task, one unique answer. Limit each task question to only one actual thing you want people to do, and one unique answer.
6. Does not contain clues. The participant will examine the task question like Sherlock Holmes would hunt for a clue. Make sure it doesn’t contain any obvious keywords that could be answered by conducting a search.
7. Short—30 words or less. Remember, the participant is seeing each task question for the first time, so aim to keep its length at less than 20 words (and definitely less than 30).
8. No change within testing period. Choose questions where the website or app is not likely to change during the testing period. Otherwise, you’re not going to be testing like with like.

References

1. https://measuringu.com/benchmark-website-usability/
2. https://www.usertesting.com/blog/2015/01/05/benchmarking/
3. http://www.usabilitybok.org/usability-benchmark
4. http://scottberkun.com/essays/27-the-art-of-usability-benchmarking/
5. https://measuringu.com/benchmark-tips/

Templates (if applicable)

1. Usability Benchmark Observation & Analysis Template

Created by: Joe Steinkamp | Last updated by: Joe Steinkamp