Here’s part 2 of my conversation with LukeW. Please see Functioning Form for part 1.
Let me try to clarify by taking a step back. I see field research as a way to remove some of the tarnish that comes with more “traditional” market research like focus groups and surveys. The common perspective is that people in a focus group or survey won’t really tell you what they want or how they do things because often times -they can’t. They are a level removed from the actual activity and as a result may leave out key details or considerations they use the make decisions.
The classic example, and I can’t recall where I first read it, is the washing machine manufacturer that polls thousands of potential customers and asks them “what features do you want in a washing machine?” The responses they get back are: “just the basics”; “i just want a simple setting for colors and whites”; “nothing too fancy”; etc. So the company makes a bunch of no-frills, feature-lite machines and they don’t sell because when it actually comes time to buy a machine the same people that said they want “simple above all else” fall prey to feature-sheen. “Oooh but this one has more features…” I’m sure you’ve heard a similar tale or two.
So what we have here is people saying they do one thing then going out and doing something totally different. Field research should ideally be there at the point of the sale -in context- to enable the company to see what really happens.
Now let’s go back to my original question about digital context. In all the methods you described above -great list by the way!- we’re asking people to tell us what they’re doing rather than being there -in context- when they are doing it.
Maybe I’m picking nits here but I know there are lots of “hidden” subtleties within digital social systems that govern how people behave. There are contexts of when and where that alter behavior. As an example, during a home visit a buyer on eBay may tell you: “I leave positive feedback when I get an item in good condition.” Their actual behavior, however, differs. They may or may not leave feedback based on the type of seller (professional or amateur), how much feedback they have, how much feedback the seller has, the category they are buying from, their intentions for the item after they get it (resell, return), and so on.
I guess when I think of people that spend hours every day immersed in something like World of Warcraft I feel there’s more to their behavior and motivations in that digital space than they can explain in words. How can we be a fly on the wall within that digital context? Or is what I’m looking for already covered by the methods you outlined?
Any situation where you have someone telling you about their own behavior is going to include some amount of bias (and let’s pretend for the sake of discussion that our own bias isn’t an additional factor). In focus groups, those influences are hard to leverage (complex peer dynamics, sterile environment, closed-ended discussion), but in contextual research, we can try to take advantage of showing and telling, for instance. Having someone walk through their previous feedback log, and explain, is illustrative of patterns that person may not explicitly be telling us.
Q: Are you leaving feedback for the seller?
A: I leave positive feedback; it’s really important, I usually will look at the condition and decide based on that.
Q: [points to computer screen]Can you walk us through some examples of feedback?
A: Sure, umm, here’s one I left last week. The item was in pretty good condition, so, well, I only left 2 stars, because I didn’t get it right away. And this one here, I left 4 stars because the last time I bought from them it was great. Yeah.
Q: What’s that icon over there?
A: Oh, there’s a bunch of items awaiting feedback.
Q: How often does that happen?
A: Well, I’ve got about 35 in there, some of them let’s see oh yeah, some of them go back about 3 months, I guess.
Now, I’ve totally made that up to support my point so let’s not treat it as data, but as a likely scenario for a dialog in an interview. So much of the process involves triangulation – asking the question at different points in the interview, getting demonstrations as well as declarative statements as well as stories. When you come out of the session, you have to ask yourselves what you think that person’s approach to feedback is. And it lies somewhere in between, but it’s ultimately an interpretive answer.
We can measure people’s TV viewing habits, let’s say, with a Nielsen box, and we can ask them about their behavior. And history shows (like the washing machine example) a disconnect. People under-report their TV watching. It’s easy to think about why; TV is bad, it’s better to show yourself as someone who reads books and goes on hikes than it is as someone who watches a ton of TV. The insight comes out around the delta between the observed and the reported. Of course, not every gulf is an insight, or one that you can use.
And maybe to make it more simple, it’s easier to talk to people about what they did in the past, and why (i.e., leaving feedback) rather than their overall attitude. People make generalized statements but the actual examples contain a lot more subtlety. This of course leads to one reason that companies like to reject any form of customer research – because people can’t talk about what they will do or what they will want. My short answer to that is that the researcher is the one doing the interpretation; in all of this we aren’t simply collecting responses verbatim – we are dynamically choosing different questions and making inferences in order to build our own model of how people will behave.
That said, I had an interesting conversation the other night with Zachary Jean Paradis, a student at the Institute of Design. He described how he had tried to do an ethnography of World of Warcraft (a MMORPG), where presumably a lot more of the behavior to be understood is taking place inside a virtual world. He felt like his traditional tools of ethnographic research didn’t hold up, and he was wishing for another month to better refine his methods. Gaming is an interesting example (and one where I don’t have a great deal of personal experience) of the online-behavior-studied-offline that we’re talking about. I’ve heard that some researchers will videotape the faces and body language of people while they are playing; I imagine you could play those back along with the matching gameplay and have people reflect on what they think was happening at the time. You can see that technique in Gimme Shelter, where Mick Jagger is watching the footage from Altamont while they interview him. A UK company called Everyday Lives does this sort of user research exclusively, preferring to passively observe, and then only interview when there’s a video record of the event to be discussed. I think it’s an interesting tool, but I think we need an ever-expanding palette of methods to deal with new situations as they emerge, rather than dogmatically rely on a single approach.
Am I still dodging your question? Or are we any closer?
Maybe I’m dodging your answer! One thing you said, though, really resonated with me “getting demonstrations as well as declarative statements as well as stories.” Since we can’t actually be a fly on the wall within complex digital systems yet -and I say that because the tracking software and log analytics software I’ve used is still a ways off from being nuanced and effective enough to match what we can do it the real world- that’s how we need to understand context: through demonstrations, declarations, stories, and of course observation of what people are actually doing on screen. Personally, I do think as digital environments become even more immersive and complex, we’ll need additional methods.
That said, let’s jump into the other topic I wanted to bring up with you. Without getting into pure semantics, why do you think a lot of ethnographic or field research is being characterized as “design research”? Is it user experience design teams within large companies trying to own the research process/data? Is it an attempt to differentiate the type of customer insights a human-centric problem solving approach can uncover from the types of insights Marketing departments have traditionally owned -like customer segmentation? Or does this type of research intrinsically belong in the “design world”?
I agree that there’s more to behavior than can be explained in words. It’s up to us to look for the deeper meanings between the words – what is said, what isn’t said, and how it’s said. As far terminology goes, I agree with your suggestion that the label can be an attempt to distinguish the methodology and/or the results from market research, and the departments that do market research. I’m so frustrated by the chaos around methodological labels. I’m sure within organizations they can create a locally-relevant nomenclature (they can, I’m not sure they do) but once you leave the boundaries of their company (through any industry discussion, conference, or online group) they end up sowing confusion. The vendors, of course, who move between companies are even more guilty. There’s a desire to differentiate from the other providers by claiming some proprietary take on doing research: Context-Based Research, PhotoEthnography, Rapid Ethnography, etc. (some of those may be actual methods claimed by actual firms; others may just be me riffing). It’s tough to balance exploring the ideas and staying “on-message” isn’t it? I guess that’s why I don’t take kindly to the terminology wars; they seem to make it more confusing for people.
So this time you did dodge my question! I’ve consistently heard “design” added to “research” when describing the type of activities we’ve been discussing. Any thoughts on the inclusion of the design label? I know you find yourself in lots of designer-focused events like Design 2.0 in San Francisco, Core77, Overlap… is there really that strong a connection between design and ethnographic research? Why doesn’t this type of research feed business models more than mock-ups? From my experience, the designers eat this kind of data up, the business folks are slow to act on it. What’s you take on that?
One suggestion is that the term is historical. Bringing the tools of ethnographic research into product development was led by a few firms that were self-described design firms (like my old company, GVO) or that had ties to design (Doblin, with their connection to the Institute of Design). I would also say (and this is a gross generalization) that market research tends to focus on the evaluative and design research tends to focus on the generative. That’s more about the goals of the research sponsors than anything inherent in the methodologies (since ethnography is ethnography).
And I think those goals or orientations that differ by discipline will affect the gusto with which they eat this stuff up. In many cases, designers are faced with tangible goals. They are committed to acting, since the product has to be designed, and is going to launch. The information they gain from research can help them solve a near-term problem (i.e., what is the organizational framework for a navigation through a space?). Even though my strategic recommendations can be as tangible as my tactical ones, they ask for actions that are much slower (i.e., launch a newsletter that addresses the transparency concerns of customers), more tangled up in organizational (same example) and resource issues (same example), and with many degrees of freedom to create a good solution (and as non-designers, that can be paralyzing).
Thanks Steve!
Tags: design research, ethnography, field research, functioning form, luke wroblewski, lukew
Specifically – does observing someone using a web site in their normal environment provide advantages over conducting research in a lab? This is a two part interview. Part 1 – http://www.lukew.com/ff/entry.asp?368 Part 2 – http://www.portigal.com/blog/design-research-a-conversation Categorie: Ergonomics Infotheek | 11-07-2006 | 21:04 | Link | Reacties (0)
Pingback by Weblog van de infotheek Human Factors met internet tips - Infotheek Human Factors 07.11.06 @ 7:59 pm
07.06.2006 by LukeW Part two of my conversation with Steve Portigal about context within digital products and the connection between ethnographic research and design continues on Steves site today: Design Research: A Conversation With Steve Portigal (Part 2) On a related note, the summer 2006 issue of the IDSA quarterly, Innovation, published a set of articles that tackle some of the same issues Steve and I discussed:
Pingback by Functioning Form - Interface Design Blog (Weblog) 07.10.06 @ 4:13 am
during a home visit a buyer on eBay may tell you: “I leave positive feedback when I get an item in good condition.” Their actual behavior, however, differs.” This is a two part interview. Part 1 – http://www.lukew.com/ff/entry.asp?368 Part 2 – http://www.portigal.com/blog/design-research-a-conversation-with-steve-portigal-pt-2
Pingback by IDSA - Human Factors Professional Interest Section 07.07.06 @ 11:47 pm
(Via Steve Portigal) Full Article 6:57 AM Permalink 0 comments Top
Pingback by Usability In The NewsUsability In The News 07.07.06 @ 2:01 pmLuke, your argument that when we are viewing users who are conducting tasks on their computer we aren’t in their context made me think…
Luke: “Now let’s go back to my original question about digital context. In all the methods you described above -great list by the way!- we’re asking people to tell us what they’re doing rather than being there -in context- when they are doing it.”
One could argue that if we are standing next to them in front of the computer, we are experiencing everything in the same context that they are: viewing images on a screen that are manipulated by buttons on the keyboard and mouse. The real challenge is determining is the mental model that users are creating to justify and plan their actions. This, I feel, is (as?) hard to do with physical products.
To help figure out these mental models, sometimes it helps to get outside of the context of the user and into the context of a particular issue. For example, if you were interested in the buying decision when it comes to washers, it may be faster and more practical (cheaper) to learn about the process by centering your study on the buying decision rather than a particular user. Analyzing and comparing several similar behaviors often makes patterns in the data pop. In that case, physically, you may go to a Best Buy and watch/interview multiple people in the appliance section.
I feel like *that* is really what you were looking for: not the context of the user but another perspective from which the actions of multiple users can be viewed to provide context and insight.
Or am I misunderstanding you?
Comment by Onny Chatterjee 07.07.06 @ 1:08 pmOnny – good thoughts – I won’t speak for Luke (and hope that he has a chance to respond as well) but I think you’ve hit on another good technique – find an offline behavior that is an analog for the online behavior.
I’ve spoken with one of Luke’s colleagues at Yahoo! who does user research for their dating “property” (love that word) – they don’t study online dating; they study dating, and they design to that.
But there are many new activities that are created for the first time via online experiences that have no offline analog. Search, for example. There are bits and pieces of search in the rest of our lives, but really, it’s a new and unique problem solving (constructed broadly) activity.
If you accept that, then the understanding-of-online-behaviors problems becomes just that much more relevant.
Comment by Steve Portigal 07.07.06 @ 1:23 pmMore than picking an offline analogy for an online behavior, I think the analogy of a different point of view in the physical world needs to be recreated in the online world.
To analyze how people move through a space in the physical world, we may sit in the corner of a room and map peoples’ movement through it. On online analogy of this is maps that show how people move through a website by clicking on different links (e.g. Ben Fry’s Anemone http://acg.media.mit.edu/people/fry/anemone/). Ben created a tool to give him a view of the online world analogous to movement mapping in the physical world.
What I’m suggesting is the creation of similar tools that give Luke and his colleagues a point of view analogous to standing in the store watching appliance buyers (or daters or whatever). What I think may be helpful to Luke is not a “technique” but rather the ability to see things from a differnent perspective.
Comment by Onny Chatterjee 07.07.06 @ 2:23 pmWith regard to the semantics of “Design Research”, I make the following differentiaton:
-Design Research refers broadly to researching any aspect of design. This certainly can include user research, but can also refer to materials research, product marketing research,design processes, etc.
-User Research, of course, is specific to the study of human behavior and physiology as it relates to to design, esp. user interfaces.
Comment by Rob Tannen 07.07.06 @ 4:36 pmOnny, thanks for the comments. A few clarifications:
“One could argue that if we are standing next to them in front of the computer, we are experiencing everything in the same context that they are: viewing images on a screen that are manipulated by buttons on the keyboard and mouse.”
-For a set point in time yes. But once the 1-2 hour interview ends and the researcher goes home, people still engage with digital communities. As anyone thats become absorbed in what they are doing on a computer knows, after a while the mouse, keyboard, and screen recede and the content or activity on the screen takes over. You loose yourself in what you are doing. This is particularly true in highly immersive, social applications.
“not the context of the user but another perspective from which the actions of multiple users can be viewed to provide context and insight.”
-That sounds right. I’m looking for how the presence or absence of content, actions, and other people in a digital environment influences behavior. It’s the digital equivalent of going to the airport baggage claim to see how you could design better luggage. You watch how people, airport, and baggage intersect and interact and hopefully glean insights for making the experience better.
Rob,
Comment by LukeW 07.08.06 @ 9:49 amSo why not “Product Research” or “Behavioral Research”. Can’t field studies (like the kind we discussed above) help set the course of a business model or marketing campaign?
We can define “design research” as distinct from “market research” quite easily: Design research is concerned with desirability, usability, usefulness, ergonomics, trust (wait, this is all in peter’s honeycomb diagram). Design research doesn’t care about the economic and emotional factors going into whether or not a consumer can be compelled to buy a product, focusing only on how the product is used — which can include emotional and even economic factors. Design research is not concerned with “conversion”. Market research, on the other hand, is almost wholly concerned with that moment where the information available about a product (the design, the advertising, the news, the buzz) meets the cold hard facts of individual lives (personal financial situations, personal and family needs and desires, self-image and psychology, etc) to induce a decision to part with their money (or their time, in the case of free products and services like TV).
Comment by Christopher Fahey 07.08.06 @ 9:54 amThis discussion of labels is hilarious and silly, as I’ve long maintained. A problem has many facets to be tackled, and we work hard to help understand as many aspects of those as we can, in order to propose the right kind of solutions. In each situation, there are different facets (i.e., conversion, ergonomics) that might be out of scope, or less relevant, or whatever. And every exploration can’t attempt to understand every facet – we must prioritize.
But there’s little practical in our work in deciding ahead of time if you are going to do market research and try to understand adoption barriers, or forget about those issues entirely and only focus on the usability of a system. It’s not effective to create such silos of approaches and desired outcomes and then tackle problems with the associated blinders on.
We try to help our clients take the right business actions (the range of which is enormous) and inform those decisions through the perspectives of their “customers” (the definition of which will certainly vary by situation, from internal to external, etc.). We’re not taking on any labels that exclude us permanently from a certain type of recommendations.
Comment by Steve Portigal 07.09.06 @ 11:49 amOh, I agree completely. Similarly, one shouldn’t rule out someone who calls themselves a “design researcher” to help you do market research, or vice versa. I was just pointing out that one can narrow one’s research focus on one thing (like, say, conversion), with a great degree of exclusion of another (like, say, ergonomics).
Comment by Christopher Fahey 07.12.06 @ 9:28 am
m obviously going over a lot of ground you are familiar with above, but maybe at this point you could clarify what you’re pushing at in terms of getting into these digital contexts, and I’ll try and be more relevant in my response! Continued Read part two of this conversation on Steve’s site. Tags: research, context, web2.0, design organizations Originally from Functioning Form: Interface Design by LukeW reBlogged by michael on Dec 31, 1969, 11:59PM
Pingback by digital aesthetics 07.13.06 @ 5:43 amSteve does a fine job of describing the practice and underlying rationale of ethnography. However, faced with his encyclopedic knowledge, readers may get the idea that you have to study everything to learn anything. Not so. In fact, an experienced ethnographer like Steve learns to see patterns in behavior and regularity in what clients may be looking for. Putting two and two together, he can then impose constraints on his research that enable him and his clients to more quickly get to the truth. This is important to know, because otherwise — and I have heard this more than once — ethnography can be dismissed as an expensive, even luxurious foray into social science. Ethnography can be thorough to a fault or more likely, intelligently abridged; but it certainly is not a luxury!
Comment by Bob Jacobson 07.26.06 @ 3:34 amBob – the “luxury” rebuff is an interesting one; I think it’s a scary process for many because it is somewhat unstructured and of course unfamiliar. I don’t think anyone disputes the need for the results, but the fears about the perceived time, perceived risk, perceived cost, etc. are daunting to overcome.
Comment by Steve Portigal 07.28.06 @ 3:38 pm