KS&R Blogs


I was so bummed when my toaster broke last week! (My favorite snack is toasted whole wheat bread generously topped with cinnamon and sugar - and yes, I realize that sugar negates the health benefits of whole wheat, but I don't care.)

So I headed out to shop for a replacement, thinking that this would be a quick and easy purchase.

KitchenAid KMT4115CU ($89.99) stopped me in my tracks. The tagline on the package boasted, "Toasting made easy." What? Had I misjudged the complexity of toasting? Perhaps my old toaster wasn't broken after all... Maybe the functions are just too ambiguous or complicated and, as a result, I am unable to operate it correctly.

Achieving the optimum UX (User Experience) means more than just "easy to use". The multi-faceted characteristics of UX can be summed up in the "5 E's": efficient, effective, engaging, error tolerant, and easy to learn. Ultimately, it requires a delicate balance of innovation and intuitive. With household appliances, for example, we look for products that anticipate our needs, make our lives better, and accomplish the task in a way that makes sense and is involving or interesting.

UX testing is an essential component of bringing a new product to market. It provides critical feedback on how real users interact with the product to uncover any problems or barriers before it's too late to do anything about it. It is also invaluable in identifying areas of improvement.

Many techniques have been developed for UX testing. In a study KS&R conducted on prototype consumer electronics for a major global manufacturer, we employed observational and task-based research, using a simple framework consisting of two main components:

  • product and solution measures -- the interface features of the product as design variables

  • environmental measures, including the end user; product; activity; and environment as context variables

How can you get the most out of your UX testing? Here are some important guidelines:

  1. Set aside a few minutes at the outset of the session for warm up. Discuss respondent needs, wants and expectations for the product, including what the UX would be like. And give respondents a few minutes to familiarize themselves with the product before beginning the tasks. This will go a long way to relieving any respondent anxiety --- and reassure them that they are not the ones being tested.

  2. Develop a set of structured tasks that are essential to the product's success. What must a user be able to do? Make sure that the tasks are realistic and reflect the ways in which users will actually use the product.

  3. Explore respondents' expectations about how difficult they expect each task to be. The Usability Magnitude Estimation (UME) is a measure that assesses respondents' expectations of the ease / difficulty of a task. Respondents rate how difficult a task is likely to be before they try to do it, and then provide a second rating after trying to complete the task. As a result of these ratings, each task is ultimately assigned to one of the following categories:

    • Tasks that were expected to be easy, but were actually difficult

    • Tasks that were expected to be difficult, but were actually easy

    • Tasks that were expected to be easy and were actually easy

    • Tasks that were expected to be difficult and were difficult

    Be sure to "deep dive" and pay particular attention to tasks that were expected to be easy, but were difficult to complete; these will be a priority for the product design team.

  4. Make the tasks part of scenarios rather than direct instructions. Respondents perform much more naturally when confronted with situations they can readily connect with. "Imagine that you have unexpected dinner guests and need to quickly defrost a roast in the oven" is much better than "Find the defrost button and set the timer for 20 minutes."

  5. Assign respondents only one task at a time. If there is a complex task that respondents need to undertake, break it up into smaller, individual tasks. Overloading respondents can confuse them, or muddle their approach to the task.

  6. Verbally walk respondents through each task to be completed. Ask them to repeat back to you what they understand the task to be. Then provide them with a written overview of the task details as a reference. Be careful with how you describe the task – you don't want to inadvertently lead respondents. And be sure to politely decline any requests for explanations or assistance.

  7. Make the most of the opportunities to ask questions! Solicit respondents' feedback while they perform each task ("think aloud") as well as after completing all tasks (debriefing retrospectively). Reacting to the experience as it happens results in fresh, rich insights in a way no other method can deliver. And the debriefing gives respondents an opportunity to react to the overall experience, including offering recommendations on how the experience could have been improved.

Lynnette S. Van Dyke
Principal, Founding Partner

Lynne Van Dyke

Lynne leads KS&R's Qualitative Center of Excellence, and specializes in brand imagery, creative ideation and motivational research applied to marketing. She is a nationally recognized Master Moderator, and has conducted more than 3,500 focus groups and interviewed more than 40,000 people in groups and individually. Lynne holds an MA and BA from Ohio University, and certification in education. She has extensive training in group dynamics and projective methods. Lynne is known for applying creative research techniques to identify and deliver the “big ideas” for brand development and strategic planning.