The first episode of Working (a new podcast) features Stephen Colbert explaining in great detail the process of creating The Colbert Report. The entire episode (listen here) is really good process stuff (creativity, collaboration, finding the story, media firehose, working under pressure) but I want to call out the section about how he prepares and uses the questions for interviewing his guest, as it’s is quite consistent with what I wrote in Interviewing Users.

And then I read the two sheets of questions that the writers have come up, what their ideas are. I usually pick 10 or 15 of those. But I don’t look at them. I don’t look at them until right before I go over [to the set], and then I read them over once again in front of my producers to get a sense of, oh, this is how my character feels about this person.

Come show time…I take them out and I go, oh, yes, these are the questions I chose. And then I try to forget them and I try to never look at the cards. I just have a sense in my head of how I feel. And the cards are in front of me, but I try not to look at them at all. I’m pretty good. Maybe I look once a week at the cards. I put my hand on them, so I know I have them if something terrible happens, but as long as I know what my first question is for the guest I kind of know what every other question is, because I really want to react to what their reaction to my first question is.

And I usually end up using four of the 15, and the rest of it is, what is the person just saying to me? Which makes that the most enjoyable part of the show for me. Because I started off as an improviser. I’m not a standup. I didn’t start off as a writer, I learned to write through improvisation, and so that’s the part of the show that can most surprise me. The written part of the show, I know I can get wrong. You can’t really get the interview “wrong.”

McRoskey mattress jumping is serious work [SFGate] – Silly-but-true stories of product manufacturing. I guess if feet are good enough for grapes they are good enough for mattresses.

Jumping on a mattress is one of the final steps in making a handmade mattress. It may be true that machines, which can be made to do most things, can be made to jump on a mattress. But a machine cannot do what Reynoso and his toes can do, which is to expertly compress no fewer than 28 layers of fluffy cotton batting while seeking to detect pea-size mattress lumps or other imperfections, the kind that can give insomnia to fairy-tale princesses and real-world princesses, too. Reynoso does his jumping in the McRoskey mattress factory on Potrero Hill. McRoskey has been stomping out high-end mattresses in San Francisco for 112 years and is something of a cult among mattress fanciers.

The yard. [Marcin Wichary] – Field research sometimes gets us backstage into interesting environments where we can ask questions and get all the details about how something works. And so I loved this tour of a bus yard, filled with great photos of artifacts, processes, signs, and interfaces.

My friend showed me around the MUNI Kirkland bus yard. MUNI is the municipal public transit system serving the city and county of San Francisco. It will turn exactly 100 later this year. The Kirkland bus yard, near Pier 39, is one of the smallest and oldest bus yards in San Francisco. It is dedicated solely to diesel buses running mostly neighbouring lines, and some express routes too. There are typically over one hundred buses leaving this yard every weekday morning for the rush hour; I visited on the weekend, when it was much quieter and many of the buses were still on the site.

To Have the Most Impact, Ask the Right Questions [HBR] – I’ve written about seventeen types of interviewing questions; here’s another simpler framework that isn’t focused specifically on interviewing.

1. Convergent questions: What, where, who, and when questions get a person to clarify the specifics of what he or she is thinking. Converging questions can be important when time is of the essence or you are dealing with someone who is theoretical.
2. Divergent or expansive questions: Why and what if questions ask a person to expand on what he or she is thinking. Divergent questions can be important when you need someone to see the larger context of a position.
3. Integrating questions: If…then what questions demonstrate an attempt to find common ground between opposing positions. This builds trust and encourages compromise, which is important in situations where the stakes are high for both sides.

Architect Bjarke Ingels’s Youthful Ambition [New Yorker] – Here’s a principle from improv applied to a fresh context: managing creativity and vision in an architectural firm.

“I think you can have high competence, ambitious, without having stress and fear as the motivating factor. It’s one of the ideas of [his manifesto] Yes is More: you can be critical through affirmation rather than negation. You can be critical by putting forward alternatives rather than spending all your energy whining about the alternatives you don’t like.”

For a little Friday Fun, here’s Mike Birbiglia‘s new short film from This American Life LIVE (if you are in the US, Canada, or Australia, I highly encourage you to find a screening near you for this next Tuesday; truly a wonderful entertainment and storytelling experience).

In this short and gently comedic film, Birbiglia pokes fun at some the norms of interviewing (and being interviewed).

I’m cited in Developing Your Interviewing Skills, Part I: Preparing for an Interview, with a set of question types. The article suggests those question types are helpful in preparing an interview guide. I think they are also very helpful in the interview itself, as you will often have to probe a number of different ways to get at what you are think is interesting.

Anyway, I’m not sure where the author found that set of questions, but I’ve recently rewritten and restructured them for the book. This seemed like a great opportunity to share them with everyone. I’d love your feedback: What am I missing? Do you disagree? What else would be more helpful for readers?

Questions to gather context and collect details

• Ask about sequence “Describe a typical workday. What do you do when you first sit down at your station?-Then what do you do next?”
• Ask about quantity “How many files would you delete when that happens?”
• Ask for specific examples “What is the last movie that you streamed?” – Compare this to “What movies do you stream?” The specific is easier to answer than the general and becomes a platform for follow up questions.
• Ask for the complete list “What are all the different apps you have installed on your smartphone?” – This will require a series of follow up questions, e.g., “What else?” because few people will be able to generate an entire list of something with some prompting.
• Ask about relationships “How do you work with new vendors?” – This general question is especially appropriate when you don’t even know enough to ask a specific question (e.g. in comparison to the earlier example about streaming movies). Better to start general than to be presumptive with a too-specific question.
• Ask about organizational structure “Who does that department report to?”

Questions to probe on what’s unsaid

• Ask for clarification “When you refer to “that” you are talking about the newest server, right?”
• Ask about code words/native language “Why do you call it the ‘Batcave?'”
• Ask about emotional cues “Why do you laugh when you mention ‘Best Buy?'”
• Ask why “I’ve tried to get my boss to adopt this format, but she just won’t do it-” “Why do you think she hasn’t?”
• Probe delicately “You mentioned a difficult situation that changed your usage. Can you tell us what that situation was?”
• Probe without presuming “Some people have very negative feelings about Twitter, while others don’t. What is your take?” – Rather than the direct “What do you think about Twitter?” or “Do you like Twitter?” this question introduces options that aren’t tied to the interviewer or the interviewee.
• Explain to an outsider “Let’s say that I’ve just arrived here from another decade, how would you explain to me the difference between smartphones and tablets?”
• Teach another “If you had to ask your daughter to operate your system, how would you explain it to her?”

Questions that create contrasts in order uncover frameworks and mental models

• Compare processes “What’s the difference between sending your response by fax, mail or email?”
• Compare to others “Do the other coaches also do it that way?”
• Compare across time “How have your family photo activities changed in the past five years? How do you think they will be different give years from now?” – The second question is not intended to capture an accurate prediction. Rather, the question serves to break free from what exists now and envision possibilities that may emerge down the road. Identify the appropriately large time horizon (a year? Five years? Ten years?) that will help people to think beyond incremental change.

Inspired by the passionate critiques I read at Authentical, here’s mine. Today, a horrific experience establishing an online account with a State of California website. Although creating a new account is almost an automatic activity at this point, I had to try 5 times to create both a username (which had to have letters and a number, and be between 8 and 12 characters) and a password (which had to have letters both capital and lowercase and a number, etc.) that would work. I’m not sure how that ended up being hard for me, but it did.

But the hysterical part was the security questions. This site required me to set up answers to four security questions. My use case for the security questions is for those situations where I can’t remember which particular configuration of password I used and I need to get a reminder or reset it. Isn’t that everyone‘s case? So we need the reminders to be unambiguous. Fact-y type things like the standby Mother’s Maiden Name, or first pet’s name, etc. are pretty common. Obviously, if they are unambiguous, they can be broken. Somewhere someone can find out your first pet’s name. It won’t change. It’s objective.

These questions are much more personal and I suppose thus are less easily divined by an intruder. But the answers are far from immutable. I had absolutely no confidence I could come up with four questions that I would answer the same way 100% of the time. Even if I could fake out my future password-forgetting self by agreeing with him that I would say the Rolling Stones are my favorite band despite regardless of any wavering in my fandom, I couldn’t successfully negotiate the dialog. What was my dream job as a kid? Well, at one point it was stuntman, then actor, then writer, and I think even director (let’s leave the armchair shrink out of this for now, shall we?). If I put stuntman now, what will I remember when I forget my password?

The Four Questions

Taking those sets of questions away from the context of the registration process, I find them quite creepy, evoking some intimacy that doesn’t exist between me and the government website, or those Facebook memes cum virii where your friends exhort you to answer a random set of personal questions and then get other people to do the same.

Note: there are some wonderful satirical examples of bad security questions on Twitter under #BankSecurityQuestionsIdLikeToSee.

Fieldwork transcripts are divvied up among team members

Over a few days spent reviewing interview transcripts from the US and China, I was struck by the observations around storytelling vs. listening vs. followup questions in Returning to America from a life in China (abstract only)

[I]n the States, I often had trouble responding to personal stories. But soon I realized that it didn’t make much difference what I said. Many Americans were great talkers, but they didn’t like to listen. If I told somebody in a small town that I had lived overseas for fifteen years, the initial response was invariably the same: “Were you in the military?” After that, people had few questions. Leslie and I learned that the most effective way to kill our end of a conversation was to say that we were writers who had lived in China for more than a decade.

At the times, the lack of curiosity depressed me. I remembered all those questions in China, where even uneducated people wanted to hear something about the outside world, and I wondered why Americans weren’t the same. But it was also true that many Chinese had impressed me as virtually uninterested in themselves or their communities. That was one of the main contrasts with Americans, who constantly created stories about themselves and the places where they lived. In a small town, people asked very little of an outsider – really, all you had to do was listen.

The 2008 documentary Tyson by James Toback is a compelling and revealing work. From a technical perspective, it’s a fun watch because Toback experiments with visual fragmenting and layered storytelling styles. In terms of subject matter, one would be hard-pressed to find a juicier, more tabloid-soaked figure to focus on, especially for those of us who came of age in the 80s. I walked away from the film with a much more nuanced and complex, though still ambivalent, view of Mike Tyson as a powerhouse boxer, as a convergent cultural figure, and, finally, as a very complicated human being.

But there was one moment that stood out, and it hammered home the incredible power of a simple interviewing technique: silence. At one point about mid-way through the film, Tyson was yammering in a very straightforward way about the fact that his desire to box and dominate stemmed from his being bullied as a young boy (predictable!). Toback must have sensed something simmering just below the surface, because when Tyson finished this train of thought Toback just let it sit. And sit. And sit. As the audience sits. And sits. Until Tyson looks back up with a completely different expression, almost with a different personality, and bares the real, brutal truth. It’s a moment when time kind of stops; I gasped out loud. It’s this kind of thrilling moment that we experience in our best interviews, when the person (“consumer!”) goes beyond just citing facts or recounting stories, to communicating to us, and our clients, something surprising, something of real value and meaning.

If you liked this interview tip, you’ll love this: Steve will be talking about his interviewing secrets at the UIE virtual seminar on the 28th of this month!

• Forrester’s 2010 Customer Experience Rankings [Customer Experience Matters] – # Retailers take 12 out of the top 20 spots. Most of the top rated companies on the list are retailers. Hotels also grabbed three of the top 20 spots. Interestingly, three financial services firms also cracked the top 20: credit unions, SunTrust Bank, and Vanguard.
  # Healthcare, Internet and TV services dominate the bottom. The bottom 11 companies on the list came from only four industries: five health insurance plans (United Healthcare, Medicaid, Anthem, and CIGNA), three ISPs (Charter Communications, Comcast, and Qwest), two TV service providers (Charter Communications and Comcast), and one credit card provider (HSBC).
• Can Design Change Behavior? [Stanford School of Engineering] – Because behavior can be influenced—not just observed—it provides an important opportunity for tackling complex challenges such as sustainability. That opportunity is perhaps best addressed with design…With this outlook, Banerjee says he is excited to be one of the principal investigators in a new project funded by the U.S. Department of Energy in which he is working with other Stanford professors who have expertise in behavioral sciences, communications, human-computer interaction, and behavioral economics. The team aims to create interventions that influence behavior to bring about significant reductions in energy use. But what designers understand well is that people are “predictably irrational” and influenced by emotional as well as rational criteria, Banerjee says.
• The Art of Asking the Question [UIE Brain Sparks] – Steve Portigal will show your team the art of asking the question. You might visit the user in their office or home, have them come to you for a usability test, or even have a chance encounter at a trade show or while waiting for an airplane. Do you know what to ask? Do you know what to listen for, to extract the critical detail of what they can tell you about your design?

  Steve will help you prepare your team for any opportunity, be it formal user research or less structured, ad-hoc research. He’ll also give you tips on how to work with your stakeholders and executives, who may also be meeting potential customers and users, so they know what to ask and how to listen—integrating their efforts into the research team. (Wouldn’t it be great if they understood why you’re doing what you’re doing?)

  Update: Use promotion code CHITTAHCHATTAH to get lifetime free access to the recording after the fact (normally a separate cost)

On January 28, I’ll be presenting a UIE Virtual Seminar entitled Deep Dive Interviewing Secrets.

Steve Portigal will show your team the art of asking the question. You might visit the user in their office or home, have them come to you for a usability test, or even have a chance encounter at a trade show or while waiting for an airplane. Do you know what to ask? Do you know what to listen for, to extract the critical detail of what they can tell you about your design?

Steve will help you prepare your team for any opportunity, be it formal user research or less structured, ad-hoc research. He’ll also give you tips on how to work with your stakeholders and executives, who may also be meeting potential customers and users, so they know what to ask and how to listen-integrating their efforts into the research team. (Wouldn’t it be great if they understood why you’re doing what you’re doing?)

I’ve also put together this quick preview to get you more of a sense of what I’m going to cover.

Sign up here for this informative event!

Update: Use promotion code CHITTAHCHATTAH to get lifetime free access to the recording after the fact (normally a separate cost)

• Six Questions from Kicker – The folks at Kicker Studio are running a series of mini-interviews with smart and accomplished design/thinky types

As I’ve said before, garbage in, garbage out. From Rob Walker’s Consumed

Recently, Stardoll did a study of its own, polling United States users about their brand preferences. Apparently they saw real-world brands on the same plane as the half-dozen or so invented brands that exist only within the site. (Some respondents even made the – clearly impossible – claim that they wear the strictly digital Goth-style brand Fallen Angel to school.)

These sorts of stories always crop up in market research and business case studies. And they are wonderful because they illustrate the depth of meaning the products, services, brands, and stories we create can be to the people that consume them. So meaningful that they will conflate pretend brands online and tangible experiences offline. Wow, we marvel, that tells you how great our stuff is; they will lie about it.

But the flip side to that is that if you are going to ask people what they think and do and want, you better have a way of triangulating their responses against other data. If you don’t know more about the person than their response, how can you contextualize it? If you don’t know what they are really saying when they answer the question – if they understood the question or are answering it in the way you intended – then you must be very careful in what you conclude and how you act on those answers.

From our Design Research Methods class, some observations from an interviewing exercise.

The scenario was to conduct interviews in order to uncover opportunities in helping people to manage food, meals, nutrition, etc.

The question asked was
What are the challenges you face in meal preparation?

Of course, that question is flawed because it presumes that there are indeed challenges. This was evident when the respondent struggled with how to answer outside the frame of the question.

An alternative might be
Are there any challenges in meal preparation?
which is is more open-ended.

But better still is
What are your feelings about the experience of meal-preparation?
since it doesn’t put the label challenges into play. It would be important to understand the labels the person being interviewed places on the different aspects of their experience, and to use their terminology to probe further.

Also worth noting is that the original question came right off the sample interview guide I distributed. Sometimes the interview guide is a tool to document “questions you want answers to” rather than “questions you want to ask”; doing fieldwork involves a lot of translation back and forth between the two.