March 17, 2009

Lies, Damned Lies and User Research

At the Funologists panel at SXSW , the speakers lightly touched on digital ethnography. One of the points they that came up was the “lying user” phenomenon. While they didn’t go into it too far, it sparked an idea I’ve been meaning to write about.

One of the hardest parts about user research and analysis is controlling for lying users. It’s not that users try to be deceitful. But there are many reasons why they may not be telling you the truth. Here are four:

Need to represent themselves.

Users may lie about choices they have made in the past because the truth reflects negatively on their self image. For example, a user may state they purchased an iPod because of it’s ease-of-use, when the real reason was that they felt buying a SanDisk MP3 player would mark them as less hip.

For this phenomenon, a best practice is to probe further until you can get specific information about what influenced the decision.

Inability to forecast future actions.

People are notoriously bad predictors of how they will behave in the future. They expect their future behavior to be markedly different than what past behavior suggests. Similarly, their ability to recall previous actions degrades over time. For example, a user may say that they really want a customization feature. If you ask them to recall customizing a similar feature in the past 6 months, the answer is often very different. In both cases, the further you get away from the present, the rosier the glasses become.

A tip for handling this situation is to follow questions that speculate about behavior with “grounding” questions that ask them to recall behaving like this in the past.

Fear of insulting the moderator.

Simply to avoid being critical or confrontational, users will often not truly confess dissatisfaction with a product. Not wanting to hurt the feelings of the researcher is a common unspoken issue.

At the onset of research, the researcher should explicitly distance himself from the design, saying something like “Be candid. I’m not the designer, so nothing you can say will hurt my feelings.”

Fear of looking uncreative.

This happens often in sessions where you simply ask the user to rattle off features they would like. Often, users base their responses on those they already know exist–rather than suggesting features that solve real problems they face. For example, you may hear “I’d like the page to be customizable like iGoogle” when they have no precedent of customizing pages.

For this situation, it’s ok to solicit features but make this a low-priority research tactic. Instead, focus on the actual challenges they face, agnostic of a solution.

Wrap up

These are just four of the many reasons users lie to researchers. The message is this: when doing research, consider the answers they give but be cautious and critical. Don’t take any self-reported information too seriously, since they are easily tainted. Where you expect there to be lying bias, probe for detail and past history to validate their responses.