Do as they do, not as they say
August 7, 2001
While I think most of what Jakob Nielsen has to say is more appropriate for conference speaking appearances than for actually building sites or products, his latest column hits the nail on the head. Unfortunately, too many people will take the title, "First Rule of Usability? Don't Listen to Users," too literally without understanding his real point: don't believe what users say they do (or will do), instead you need to watch what they actually do.

Although he doesn't say it directly, Nielsen's column is more about the misuse of surveys and focus groups—often driven by marketing people who are familiar with those techniques. Unfortunately, having spent a summer in college as a field interviewer for a market research company, I'd say even when used for appropriate purposes that most surveys and questionnaires are so poorly designed/run that you often get back lots of bogus data.

For example, I recently was recently called and asked to take part in a survey to find out why I'd dropped my old cell phone provider. The survey asked me to rate various services—but had no way to put a "not applicable" for services I'd never bothered to use… Needless to say the information from those answers is worthless.

But the bigger issue is that surveys and focus groups are an inappropriate substitute for actually observing users. Which is why the user research we do is more like clinical diagnosis. It's less directly quantifiable, it's more expensive because it's leans toward being one-on-one, and relies heavily on the skill of the person doing it to reach good conclusions. But because it's based on observations, less prone to self-reporting bias. It's not as good at uncovered attitudes, but that's not our focus, our focus is on behavior.

The second issue is that data doesn't equal information, and information doesn't equal knowledge. Part of the designer's job is to figure out what users really want/need, regardless of what they say they want/need. This is a collarary to Alan Cooper's principle that you can't test your way to usability, you have to design your way to it.

(And there times when it is legitimate to not listen to (some) users. Computer game design guru Chris Crawford once observed that version 3.0 of any game would usually instantly crush any player who hadn't played versions 1.0 and 2.0. Why? Because the most vocal feedback came from hardcore gamers who wanted bigger and nastier challenges—and since the designers were usually hard-core gamers, these were more fun to design. So the games would get harder and harder, and although they might do well, eventually they'd get too hard for the larger market of buyers. Incidently, this is a good example of how using personas might have avoid the trap, assuming you had the right personas.)

As far as why business want to use questionnaires, most people are familiar with marketing research which likes to be "quantitative" and relies heavily on surveys —in part because they're cheaper than other techniques like ethnography. And surveys, done well, are a decent way at getting toward attitudes, which is marketing's main concern. Scarily, there's a trend toward using focus groups over surveys because they're even cheaper—never mind that they're even less reliable and even marketing experts recognize they should only be used as a springboard for ideas, not a source of literal truth.

There is a place for questionnaires, primarily for gathering background info about users. Market segmentation data can be helpful in developing personas, and it may be useful to survey users (particularly within a company) about skills levels, areas of knowledge, etc. Of course, you need to be skeptical of people's self-reporting on these things, since they'll often overstate their abilities.

But since most people aren't necessarily familiar with our approach, nor does it offer the "safety in numbers" of quantitative techniques, we need to be able to argue the business value of using it over the techniques they're familiar with. ::

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Copyright © 2001 All Rights Reserved.
Any problems with the site?

Note: This site will look much better in a browser that supports web standards, but it is accessible to any browser or Internet device.