Why we love to hate "user testing"

It’s become popular amongst usability professionals to exclaim “We don’t call it user testing, we’re not testing users” Although (of course!) I understand the sentiment it is often proclaimed in a fashion implying this is a truth a majority of people are not aware of.

I have trouble, though, finding people who sincerely believe user testing means finding out if the human being is smart enough to use the tool.

The assertion works especially well as a rhetorical device in presentations and sales meetings. The audience goes “Aaah!” and one can quickly move on to the next slide. I know. I’ve done it. More than once.

But in a similar fashion I have yet to find people who believe that ski jumping is a sport where participants jump over skis, or hear voices calling for a ban of the term fly fishing because people are taking their boats out to capture winged insects.

Language is funny that way, if you want to infer a different meaning or unfavorable connotation on a word pair you often can. I promise you the deadlifts I do at the gym do not involve any foul play or zombie-induced focus.

Sometimes the ambiguity of a term is actually what gets people interested in understanding more.

People in UX enjoy, and rejoice in, finding double-meanings that could cause confusion. It’s part of their job. But that overzealousness will create some false positives. Just because something can be misunderstood does not mean it often will be. Especially when it is used within a clear context.

Instead of worrying about your own interpretation of “user testing”, I would like to propose that it’s more important to follow up and ensure that people understand what you mean.

The point of the phrase user testing has undoubtedly been to make clear the fact that you are involving people who are users of the product/service you are building or maintaining. I find it hard to accept that the phrase — in the context of a building a product or service — does not communicate this. This is how it has been used for many years.

And then someone figured it was demeaning to call someone a user of a system? Really? Fine, I’ve had conversations around the possibility calling users just people. But let’s at least make the effort to differentiate between current users and potential users. That’s when the concept of current people and potential people kind of has the wrong ring to it.

To be blunt, I do not believe that people who call it usability testing are generally doing a better job than people calling it user testing so I’m going to give that whole ordeal some rest.

My rant is over, but thank you for reading this far. Now on to my real point.

All this said, I actually don’t believe any of the phrases user testing, usability testing, UX testing or similar variants really do justice to the task being performed. Maybe the word we really should be questioning is the second one: testing. Over the years this has become a checkbox in a project plan and without the proper management testing is often overlooked as a passing task that once completed can be moved to a manila folder and archived.

The word test is not something most people who have attended school have fond memories of. Speaking of connotations, tests do tend to bring some dark ones to light: midnight cramming, profuse sweating, stress, you name it.

And really: should a usability test fail or pass a system, or should it focus on bringing new knowledge to the table?

Testing, by itself, is really not the objective is it? I put it to those who find it valuable to point out that user testing is not about testing users: is it then about testing the product? I find it closer to the truth that we are observing the interplay between a user and a system. From this interplay and communication we come closer to understanding what is helpful and what is detrimental when it comes to the users, as well as the organization, reaching their goals.

What we really are looking for is not a right or wrong answer (“Does this work or not?”) but one or more collections of insights, validations and learnings that help us take the next step so that we may then design the next experiment to validate any new assumptions (“It could work better if we do this.”). Wouldn’t it be nice at a project demonstration to discourage the questions “What did we test? Did we pass?” And instead promote the more curious “What did we learn?” and “What do we now believe?”

I much more enjoy talking about my work using phrases such as why- or insights finding, expectations analysis, sense-making or behavior observation. Also, the more I can help people think about purpose and goal rather than bluntly label the task, the more curiosity I can spark with my stakeholders. It is when everyone, not just me, understands the value of the data that real change tends to more readily happen.

Enough of my fluctuating attitude though. Back to you, what is it that you do and what do you like to call it?

This post was sparked by a question posed by Mike Beasley on Twitter, and I joined in the conversation with Becca Kennedy. I was in retrospect too harsh in my first tweet using the word “snob”. I blame Twitter’s word count and my having recently been getting overly worked up about the common practice of criticizing the term “user testing”. A practice I’ve admittedly partaken in myself, as I confessed earlier. In UX (itself a pretty shady abbreviation and label) we debate terminology endlessly and I don’t see our stakeholders really getting any happier about it. Hence I just needed to get this post out of my system. Thanks Mike for adding just the right amount of kindle to my fire 😉