Question
How do you teach a non-designer to effectively critique a design?
Answer
I haven't used it myself yet, but I really like the concept behind "collage" techniques. Give users pen/paper to draw what they want, or take the existing elements of your design and make 'em cut out jigsaw puzzle pieces, along with some blank pieces, and get them to play around with the arrangements. Perhaps make it an individual or team game.
One element among the previous examples that best captures the same spirit is Xianhang's suggestion of the "if this website were a dog, what kind would it be" question. This challenges the user to think at a very high, creative, right-brained level.
One reason I think these are great is that the approach gets rid of the paternalism/nannyism that is implicit in some of the other (well-intentioned) suggestions here. Some respondents are answering in ways that suggests a "nanny/child" relationship. I also sense a certain defensiveness in the design community along the lines of "we are the creatives, you users are just the data sources that we manipulate through behavioral experiments like biologists designing mazes for rats.... just run this damn maze, don't try to do maze design."
This attitude is both mistaken and counter-productive. Designers are different from non-designers primarily in
a) being equipped with a more sophisticated vocabulary and analysis/synthesis techniques and
b) having a bigger library of reference UX examples, patterns and precedents in their heads.
They are NOT fundamentally smarter or more creative. In fact you may be dealing with users who have vastly more domain knowledge than you, and fundamentally more creative. You should use that, not fight it.
For example, though I've done a fair amount of Web design work (at the conceptual wireframe level, not detail), I am not a trained designer in the sense the other people here are. I am an aerospace engineer with a systems/control background. If we are talking about the UX of a product like Matlab that is designed for (and to a large extent, by) people like me, I am afraid I would have VERY little patience for the kind of nanny attitude some designers are prone to display. I would (correctly) believe I know a hell of a lot more about what needs to be done than you. Same holds for say finance software, tax software etc.
Similarly the customers for people like me (or people like me who are actually doing the work their degrees are about, instead of wandering off track like me) are people like pilots, who actually use control systems in the real world.
Pilots provide feedback to control system designers using terms like "the pitch response feels kinda spongy during barrel rolls" (okay I made that one up :)).
Control system designers translate that kind of feedback into things like the design of signal filters in the feedback loop. The pilot may know nothing about filter design, but he sure as hell knows a lot more about actually flying planes and what the control responsiveness needs to feel like. I'd get nowhere talking down to a pilot. They know some aerospace engineering theory, but not as much as I do. They sure know a lot more about flying planes in the real world than me. This relationship should be reflected in the interaction. Daddy knows best won't work as a starter attitude.
So the creativity of the test subject is something to be encouraged by framing the interaction as a peer-to-peer interaction among equals with different relevant and overlapping knowledge sets. Not a nanny-child interaction. You are not designing a toy or the cartoon network website. Unless you are.
You want to structure the interaction to draw out creativity and specialized domain knowledge, not just "rat in a maze who should only talk back to me in squeaks and shouldn't insult me by talking about maze design."
The user's mental model of the interaction itself may not be as disciplined as the designer's but it may actually be more creative, nuanced and sophisticated in absolute terms. Or his mental models may be based on a different experience paradigm than yours, and be "disciplined" in a very different way (there is a systematic process for designing control systems for example, and I may be evaluating the UX of something like Matlab according to the logic of that process, rather than the logic of the interface). To finish the rat-maze analogy, the rat may actually know more about mazes than the human in some sense. Maybe they are the ones running the experiments, as suggested by Douglas Adams in Hitchhiker's Guide. If you take the comedic extremism out, Adams actually has a point. And if you consider that users are actually blessed with brains as complex as yours....
One term that I haven't heard anyone use here btw, is "protocol analysis" ... a big brother to "let the user talk out loud." I've heard some people (who shall remain unnamed) talk of the "talk aloud" technique as if it were a therapist-patient kind of talk-aloud (a healer talking to the wounded). NO! This is a very different type of talk-aloud between two healthy people.
Protocol analysis is a way of doing talk alouds based explicitly on the assumption that there is a lot of deep intelligence, domain knowledge and creativity to be drawn out. It is a cognitive psychology technique rather than a design technique. Worth looking into if you want to go beyond naive "talk aloud"... PA is often used exactly for the kinds of situations I mentioned, getting high-skill subjects like pilots to explain what they know/transmit their mental models to others.
http://en.wikipedia.org/wiki/Pro...
One element among the previous examples that best captures the same spirit is Xianhang's suggestion of the "if this website were a dog, what kind would it be" question. This challenges the user to think at a very high, creative, right-brained level.
One reason I think these are great is that the approach gets rid of the paternalism/nannyism that is implicit in some of the other (well-intentioned) suggestions here. Some respondents are answering in ways that suggests a "nanny/child" relationship. I also sense a certain defensiveness in the design community along the lines of "we are the creatives, you users are just the data sources that we manipulate through behavioral experiments like biologists designing mazes for rats.... just run this damn maze, don't try to do maze design."
This attitude is both mistaken and counter-productive. Designers are different from non-designers primarily in
a) being equipped with a more sophisticated vocabulary and analysis/synthesis techniques and
b) having a bigger library of reference UX examples, patterns and precedents in their heads.
They are NOT fundamentally smarter or more creative. In fact you may be dealing with users who have vastly more domain knowledge than you, and fundamentally more creative. You should use that, not fight it.
For example, though I've done a fair amount of Web design work (at the conceptual wireframe level, not detail), I am not a trained designer in the sense the other people here are. I am an aerospace engineer with a systems/control background. If we are talking about the UX of a product like Matlab that is designed for (and to a large extent, by) people like me, I am afraid I would have VERY little patience for the kind of nanny attitude some designers are prone to display. I would (correctly) believe I know a hell of a lot more about what needs to be done than you. Same holds for say finance software, tax software etc.
Similarly the customers for people like me (or people like me who are actually doing the work their degrees are about, instead of wandering off track like me) are people like pilots, who actually use control systems in the real world.
Pilots provide feedback to control system designers using terms like "the pitch response feels kinda spongy during barrel rolls" (okay I made that one up :)).
Control system designers translate that kind of feedback into things like the design of signal filters in the feedback loop. The pilot may know nothing about filter design, but he sure as hell knows a lot more about actually flying planes and what the control responsiveness needs to feel like. I'd get nowhere talking down to a pilot. They know some aerospace engineering theory, but not as much as I do. They sure know a lot more about flying planes in the real world than me. This relationship should be reflected in the interaction. Daddy knows best won't work as a starter attitude.
So the creativity of the test subject is something to be encouraged by framing the interaction as a peer-to-peer interaction among equals with different relevant and overlapping knowledge sets. Not a nanny-child interaction. You are not designing a toy or the cartoon network website. Unless you are.
You want to structure the interaction to draw out creativity and specialized domain knowledge, not just "rat in a maze who should only talk back to me in squeaks and shouldn't insult me by talking about maze design."
The user's mental model of the interaction itself may not be as disciplined as the designer's but it may actually be more creative, nuanced and sophisticated in absolute terms. Or his mental models may be based on a different experience paradigm than yours, and be "disciplined" in a very different way (there is a systematic process for designing control systems for example, and I may be evaluating the UX of something like Matlab according to the logic of that process, rather than the logic of the interface). To finish the rat-maze analogy, the rat may actually know more about mazes than the human in some sense. Maybe they are the ones running the experiments, as suggested by Douglas Adams in Hitchhiker's Guide. If you take the comedic extremism out, Adams actually has a point. And if you consider that users are actually blessed with brains as complex as yours....
One term that I haven't heard anyone use here btw, is "protocol analysis" ... a big brother to "let the user talk out loud." I've heard some people (who shall remain unnamed) talk of the "talk aloud" technique as if it were a therapist-patient kind of talk-aloud (a healer talking to the wounded). NO! This is a very different type of talk-aloud between two healthy people.
Protocol analysis is a way of doing talk alouds based explicitly on the assumption that there is a lot of deep intelligence, domain knowledge and creativity to be drawn out. It is a cognitive psychology technique rather than a design technique. Worth looking into if you want to go beyond naive "talk aloud"... PA is often used exactly for the kinds of situations I mentioned, getting high-skill subjects like pilots to explain what they know/transmit their mental models to others.
http://en.wikipedia.org/wiki/Pro...