Design Knowledge Base

Concept Evaluation

A type of usability test that tests multiple concepts in a single session with a participant. The goal is to obtain qualitative reactions to the concepts and analyze behaviors and performance while completing tasks on the concepts

Tests can be conducted with paper, low-fidelity prototypes, or early versions of product. The prototype would contain wireframes of the design concept with several key usage scenarios supported in this version of the concept. The approach allows for a validation of the workflow and validation of initial design decisions around navigation, terminology and rough layout.

Differs from multivariate testing by having far more differences in each version.

Not to be confused with traditional Concept Testing methods.

Research Details

Research Type Sample Size Session Time
Qualitative, Behavioral & Attitudinal, Formative Small (3 - 5 participants) 30 - 60 min

Preparation Time

When to Use

  1. To evaluate if designs address primary user needs.
  2. To help choose between alternative designs.
  3. To validate if a concept is a viable alternative to an existing product.
  4. Should be used early and often in the design process to evaluate widely divergent ideas.

Assets Needed

  1. Script
    1. Recording logistics and introduction
    2. Intro Questions
    3. Defined task(s) and scenario(s)
    4. Questions per task and scenario
  2. Multiple prototypes
  3. Note Taking Log
  4. Findings Report

Optional

  1. Recruitment Outreach Template
  2. Recruitment Screener
  3. Synthesis Artifact(s)

Steps

  1. Write a script much like you would for a usability test, include pre-interview questions and task scenarios to be completed by the participant.
  2. Make sure your tasks are not leading.
  3. Prototypes must be randomized to counter priming biases caused by seeing a concept before another.
  4. Introduce yourself to the participant, give them a summary of what’s going to happen, and conduct the pre-interview.
  5. Perform the tasks on each concept, one at a time.
  6. During the task make sure the participant is speaking aloud their thoughts, feelings, and expectations.
  7. After completing or failing each task on a single concept, ask the participants thoughts
  8. Discuss all divergent concepts together with printed versions to allow the user to discuss them.
  9. Its important to not take the participant’s preference as is. Ask why the participants prefers the selected concept.
  10. Analyze and synthesize the findings.

References

  1. Effective Formative Test Reports
  2. Two Kinds of Usability Tests
  3. https://designsprintkit.withgoogle.com/methods/validate/usability-study/

Templates

  1. Research Plan
  2. Script Template

Created by: Joe Steinkamp | Last updated by: Joe Steinkamp