Advice from the field | Usability Testing

Please add your own experiences from doing usability testing. 

Joel Denning

"Don't lead the witness": The way you phrase questions has a big impact on how they are answered. Often people will answer a question knowing how you want it to be answered

"Is this good? Would you use it?" are usually unreliable questions

"Do the user testing with the people who actually use it": at my previous job, we did too much user testing with employees of the company who had an accounting background instead of with customers who were accountants paying for the software. People who use the software have "skin in the game" and are much less likely to be satisfied with fluffy descriptions and mockups of features. They often know what actually matters and tell you it bluntly ??

"Invite developers, product managers, and other stakeholders to user testing": this was very successful at my last two jobs in building group consensus on product designs. People might not always be able to come, but it's worth it to invite them.

"Lots of lukewarm yes'es is a really bad sign": if all five people give positive feedback without anything constructively critical, then that might mean we're solving the wrong problem, are talking to the wrong users, or that the UX didn't really click with them.