Design Knowledge Base

Usability Testing

Usability (or summative) testing is the practice of testing how easy a design is to use on a group of representative users. It involves observing users as they attempt to complete tasks.

Usability is defined by 5 quality components:

  1. Learnability
    • How easy is it for users to accomplish basic tasks the first time they encounter the design?
  2. Efficiency
    • Once users have learned the design, how quickly can they perform tasks?
  3. Memorability
    • When users return to the design after a period of not using it, how easily can they reestablish proficiency?
  4. Errors
    • How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
  5. Satisfaction
    • How pleasant is it to use the design?

Research Details

Research Type Sample Size Session Time
Hybrid, Behavioral, Formative Small (5 participants) 30 - 60 minutes

Preparation Time

When to Use

  1. Towards the end of the development stage
  2. To learn if participants are able to complete specified tasks successfully
  3. To identify how long it takes to complete specified tasks
  4. To find out how satisfied participants are with the product
  5. To identify changes required to improve user performance and satisfaction

Assets Needed

  1. Script
    1. Recording logistics and introduction
    2. Intro Questions
    3. Defined task(s) and scenario(s)
    4. Questions per task and scenario
  2. Higher Fidelity Prototype
  3. Note Taking Log
  4. Findings Report

Optional

  1. Recruitment Outreach Template
  2. Recruitment Screener
  3. Synthesis Artifact(s)

Steps

  1. Create test plan
    1. Purpose
    2. Problem statement/test objectives
    3. User profile
    4. Method
    5. Task list
    6. Test environment/equipment
    7. Test monitor role
    8. Evaluation measures (data to be collected)
  2. Recruit participants
    1. They should have no involvement in the design
    2. They should be representative of the target users
    3. Testing 5 participants allows you to find as many usability problems as you’d find by testing more users
  3. Conduct tests
    1. Remain neutral - you are there to watch and listen. If the participant asks you a question, reply with “what do you think?” or “I am interested in what you would do.”
    2. Do not jump in and help participants immediately and do not lead the participant.
    3. Take good notes. Note-takers should capture what the participant did in as much detail as possible as well as what they say (in their words). The better the notes are that are taken during the session, the easier the analysis will be

    4. Measure both performance and subjective (preference) metrics. People’s performance and preference do not always match. Often users will perform poorly but their subjective ratings are very high. Conversely, they may perform well but subjective ratings are very low.

    5. Performance measures include: success, time, errors, etc.
    6. Subjective measures include: user’s self reported satisfaction and comfort ratings.
  4. Analyze results
    1. Quantitative data
      1. Data to record:
        1. Success rates
        2. Task time
        3. Error rates
        4. Satisfaction rating
      2. Demographic info (optional)
      3. Task scenarios for each metric
    2. Qualitative data
      1. Record notes related to:
        1. Observations about paths participants took
        2. Problems experienced
        3. Comments/recommendations
        4. Answers to open ended questions

  5. Present findings

  6. Modify and retest

References

  1. Usability Testing at Usability.gov
  2. Task Scenarios for Usability Testing
  3. Recruiting Usability Testing Participants
  4. Running Usability Tests
  5. Quantitative vs. Qualitative Usability Testing
  6. https://www.uxbooth.com/articles/the-art-of-guerrilla-usability-testing/
  7. https://uxmyths.com/post/831431504/myth-22-usability-testing-is-expensive
  8. https://methods.18f.gov/validate/usability-testing/
  9. https://methods.18f.gov/interview-checklist/
  10. https://methods.18f.gov/usability-test-script/
  11. https://www.interaction-design.org/literature/topics/usability-testing
  12. https://www.smashingmagazine.com/2018/03/guide-user-testing/

Templates

  1. Research Plan
  2. Usability Test Observations Spreadsheet
  3. Usability Issues Assessment Template

Created by: Joe Steinkamp | Last updated by: Joe Steinkamp