Posts tagged “analysis”

Listen to Steve on the Aurelius podcast

I had a great chat with Zack Naylor about user research for the Aurelius podcast. We talked about

  • Why you should be doing [more] user research
  • How to convince your stakeholders that user research is important
  • 3 approaches to building brilliant products and features (and which one is best)
  • Convincing your stakeholders and leaders to do (more) user research
  • What is a user research process to make sure you’re learning the right things
  • The difference between research analysis and research synthesis

It’s posted here and embedded below

ChittahChattah Quickies

What It’s Like to Experience New Technology After 25 Years in Jail [Gizmodo] – From a Quora thread to a Gizmodo post (and there may be a book deal happening?) here’s a real edge-case user type, someone with almost no exposure to current technology.

Prior to my release from prison, I gave considerable thought to a technology strategy. My wife was used to using Microsoft products, but everything I’d read indicated that Apple products offered a much quicker learning curve. On the day she picked me up she handed me an iPhone 4S. During my first week of liberty, we purchased a MacBook Pro and iMac desktop system. I hoped they would all work seamlessly together. But since my wife wasn’t as comfortable with the Apple products, she insisted that I load them all up with Microsoft products so she could rescue me when I had problems. I’ve had a lot of problems coming up to speed with simple tasks like email, or synching all of my computers together. I’ve also had a problem remembering all of the passwords she assigned to me. I keep arguing that we should use only one password, but that only brings forth her arguments on the dangers of identity theft. Since I met many men in prison who served time for identity theft, I trust that my wife has a point.

The Improbable is the New Normal [The Technium] – What are the consequences of spectacle fatigue (and I don’t mean your eyes and nose feeling sore)? And what does that mean for those who intend to entertain us (say, film and television) with more traditional content? (via Kottke)

The internets are also brimming with improbable feats of performance — someone who can run up a side of a building, or slide down suburban roof tops, or stack up cups faster than you can blink. Not just humans, but pets open doors, ride scooters, and paint pictures. The improbable also includes extraordinary levels of super human achievements: people doing astonishing memory tasks, or imitating all the accents of the world. In these extreme feats we see the super in humans.

Every minute a new impossible thing is uploaded to the internet and that improbable event becomes just one of hundreds of extraordinary events that we’ll see or hear about today. The internet is like a lens which focuses the extraordinary into a beam, and that beam has become our illumination. It compresses the unlikely into a small viewable band of everyday-ness. As long as we are online – which is almost all day many days — we are illuminated by this compressed extraordinariness. It is the new normal.

To the uninformed, the increased prevalence of improbable events will make it easier to believe in impossible things. A steady diet of coincidences makes it easy to believe they are more than just coincidences, right? But to the informed, a slew of improbably events make it clear that the unlikely sequence, the outlier, the black swan event, must be part of the story.

Overthinking It subjects the popular culture to a level of scrutiny it probably doesn’t deserve – There’s a lot of things being overthought on the site. This example is one of at least two posts (first, and second) where Law and Order episodes were put into a database and then analyzed and analyzed.

Over the entire run of the show, more than a third of all the episodes ended in Guilty verdicts, while another third ended in plea bargains. 80% of episodes ended in solid wins: either Guilty verdicts, plea bargains, or implied victories. That’s not too shabby, considering that the actual NYPD has a homicide clearance rate of about 50%. (Although you have to figure Law & Order isn’t meant to represent every case these detectives investigated; in 20 seasons, I don’t think there was a single murder that didn’t result in an arrest.)

(UPDATE 12/10/12: One of the commenters on Reddit has pointed out that the “clearance rate” has nothing to do with convictions, only arrests. In that case, Law & Order’s clearance rate would be nearly 100%, since even in the rare episodes without a trial somebody usually gets arrested. I guess I’d know this stuff if I had watched The Wire.)

The Fake Shows from Arrested Development are Now Listed on Netflix [Paste] – I love seeing fake products and brands treated like real ones. Reminds me of the in-production Newsreaders, a fake news magazine TV show that originally aired as a special episode of the parody Childrens Hospital (which also had fake promos for NTSF:SD:SUV::, leading to that becoming a real show as well.

Next spring, Netflix will premiere the highly anticipated and currently in-production fourth season of Arrested Development. Along with the rights to the show come the rights to all of the shows within the show, and to tide fans over until next spring Netflix has featured fake listings of Scandalmakers, Wrench, Boyfights and more. There is no actual footage to watch-the links just take you the Arrested Development season one page- but it’s pretty funny seeing the summaries and poster photos listed on Netflix like they belong to actual shows. Also listed are Les Cousins Dangereux, Girls with Low Self-Esteem: Newport Beach, Families with Low Self-Esteem and Mock Trial with J. Reinhold.

Steve Portigal teaching “Immersive Field Research Techniques” at UI16

Join me for Immersive Field Research Techniques coming up November 7 in Boston at User Interface 16.

My session will be pretty similar to the recent Rosenfeld Media workshop in Seattle, which was pretty well received 🙂

If you haven’t registered yet, you can use the code STEVEP for $300 off the whole conference, or $50 of a single day.

I hope to see you there!

Next week – Steve’s webinar on synthesizing user research data

I’ll be presenting my webinar User Research Analysis: You’ve Done All This Research, Now What on Thursday, April 7. This webinar is based on a workshop that I’ve led in Savannah, Atlanta, Hong Kong, Portland, Vancouver, and San Francisco, among other many other places. I’m always hearing from people who are interested but who aren’t in those cities at those times, but with a webinar, anyone anywhere can participate. I’ve adapted the workshop to suit the webinar format and added a bunch of new content based on what I’ve observed working (or not working) in the workshops, not to mention some of the latest techniques we’re using in our work.

Use the promotion code PORTIGAL when you register and get free lifetime access to the webinar that you can share with everyone in your organization. (A $40 value.) If you can’t make the webinar when it happens (time zone challenges? conflicting meetings?) you can use this code when register and watch it later at your convenience.

Check out a quick preview, below

Also available: last year’s UIE Virtual Seminar on Deep Dive Interviewing Techniques.

Sign up for Steve’s upcoming webinar on User Research Analysis Techniques

On April 7, I’ll be presenting a UIE Virtual Seminar called User Research Analysis Techniques: You’ve Done All This Research, Now What?. Sign up here!

Steve will explain synthesis, or how you turn field data into insights. Simply put, Synthesis is an iterative approach to sense-making. Steve will show you that it’s about both the experience you have as a researcher gathering that data AND the rigor of processing that data. You’ll learn the steps and types of output and deliverables that we produce as we go through the process.

Steve will help you explore ideation, where turning insights into solutions actually happens. Here’s where your hard work pays off! Ideation is about creating a wide-range of possible solutions across a wider set of areas than you can act on.

Oh, and if there’s really no such thing as a bad idea, how do you benefit from the ones that feel like they are? Steve will show you the power of “bad ideas ” and how they help you get unstuck.

Check out a quick preview, below

Also available: last year’s UIE Virtual Seminar on Deep Dive Interviewing Techniques.

ChittahChattah Quickies

  • [from steve_portigal] You Too Can Be Masterful at Analyzing Data (Go Dirty) [Cheskin Added Value] – [Darrel Rhea on the importance of outliers in analysis] At some point we grow the confidence and skill to look beyond the “tidy patterns” (however useful they might be) and focus on the anomalies. We become fascinated by data that doesn’t fit the patterns, or that doesn’t support our hypothesis. What the beginner discards as noise in the data, the master focuses on. That is where the big “Ah Ha’s” are – and where the big proprietary insights come from that can drive innovation. It’s often in weird, dirty data that we make our best discoveries.
  • [from steve_portigal] Facebook’s ‘Like’ and Conspicuous Consumption [Lone Gunman] – [Agreed, but what are our expectations for outcomes of displaying our identity and values. To connect with others who share our likes? To have our likes acknowledged and even complimented? I think there's a lot more here, no doubt that social psychologists have been studying for decades] I feel that the ‘Like’ functionality is an expense-less method of conspicuous consumption: signalling your likes and brand preferences without having to actually purchase anything (we are saying “I aspire to be the type of person who likes x, y, z” or maybe more accurately “I want you to think I’m the type of person who likes x, y, z”).

ChittahChattah Quickies

  • [from steve_portigal] Report: EPA kills Chevy Volt’s 230 mpg rating [Autoblog Green] – [Thorny problem about how to give an actual rating of a car's performance when that rating is based on gasoline consumption and the car in question doesn't (really) use gasoline! The whole frame of reference for assessing the comparative economical/ecological performance of a breakthrough product is based on a slightly obsoleting technology. Craziness ensues!]
  • [from steve_portigal] How the Old Spice Videos Are Being Made [ReadWriteWeb] – A team of creatives, tech geeks, marketers and writers gathered in an undisclosed location in Portland, Oregon yesterday and produced 87 short comedic YouTube videos about Old Spice. In real time. Those videos and 74 more made so far today have now been viewed more than 4 million times and counting. The team worked for 11 hours yesterday to make 87 short videos, that's just over 7 minutes per video, not accounting for any breaks taken. Then they woke up this morning and they are still making more videos right now. Here's how it's going down. Old Spice, marketing agency Wieden + Kennedy and actor Isaiah Mustafa are collaborating on the project. The group seeded various social networks with an invitation to ask questions of Mustafa's character. Then all the responses were tracked and users who contributed interesting questions and/or were high-profile people on social networks are being responded to directly and by name in short, funny YouTube videos.
  • [from steve_portigal] Who’s Mailing What – [A very specific form of "competitive intelligence"] Every month the Who's Mailing What! Archive receives and analyzes approximately 4,000 to 5,000 pieces of direct mail in nearly 200 categories — consumer, business, fundraising, catalogs, and much more — forwarded to us from a network of correspondents around the country. Why? Because the best way to create successful direct mail is to study other company's mail to see which campaign and techniques show up again and again. If you're tracking a particular area of direct mail — you can go right to that category, see what we've received and discover: Who's mailing what, the offers, the control, the complexity of the mailing, whether there was 4-color work, sophisticated computer work, a poly envelope, a self-mailing format.

ChittahChattah Quickies

  • [from Dan_Soltzberg] Book review: In the Loop – Knitting Now [we make money not art] – [Another form of handheld diversion – knitting. No chargers or connections necessary. Check out Mark Newport’s wonderful superhero pix halfway down the page.] In the Loop shows the different aspects of contemporary knitting practice and transforms our understanding of knitting away from retro hobby to mainstream craft and artform.
  • [from steve_portigal] An Evolution In The Data Collected By Economists [ABC News] – [Recalling the adage "You manage what you measure"] The US is deluged with economic data, yet figures cannot conclusively answer even the most fundamental questions. A handful of data-loving economists are pushing for alternative measures to provide a clearer picture of how well the economy is working. No one is talking about jettisoning the GDP, the broadest measure of the nation's economic output. By combining that information with deeper understanding of how people live, work and feel, officials hope to identify economic trouble spots more quickly and make better policy decisions. Two new sets of statistics are due to be launched next year. The Labor Department is working on an enhanced time use study to track what Americans do all day and how they feel about those activities, a project that draws on Krueger's academic research. The Commerce Department is planning a new poverty metric it hopes will provide a more up-to-date measure of which groups are struggling to meet basic needs.

ChittahChattah Quickies

  • Segmenting the Hendrix fan [] – “We believe that there is a Jimi Hendrix fan out there at 99 cents and at $9 and at $20 — all the way across the spectrum,” Mr. Block said. “We want to make each fan an appropriate offering. Is the complete Jimi Hendrix on vinyl something every music fan would want? Absolutely not. Would there be a market for it? Absolutely.”
  • Jerry Seinfeld on ideas [] – Whatever happens to “The Marriage Ref,” Mr. Seinfeld said that he was out of ideas now. “Ideas are a terrible obligation,” he said. “Who needs something else to take care of? I have kids. I’d rather nurture them than another idea.”
  • The Disposable Film Festival – In recent years a new kind of film has emerged: The Disposable Film. It has been made possible by new media (webcams, point and shoot digital cameras, cell phones, screen capture software, and one time use digital video cameras) and the rise of online distribution (YouTube, Google, MySpace, etc.). These films are often made quickly, casually, and sometimes even unintentionally. Everyone has become a Disposable Filmmaker: directors of Saturday night cell phone videos, actors under the eyes of security cameras, and narrators before their webcams. Let's face it – we live in an age of disposable film. Now it's time to do something creative with it.
  • How to Kill Innovation: Keep Asking Questions – Scott Anthony [Harvard Business Review] – Resource-rich companies have the "luxury" of researching and researching problems. That can be a huge benefit in known markets where precision matters. But it can be a huge deficit in unknown markets where precision is impossible and attempts to create it through analysis are quixotic. Entrepreneurs don't have the luxury of asking "What about…" questions, and in disruptive circumstances that works in their favor.

    So what's the alternative? Substitute early action for never-ending analysis. Figure out the quickest, cheapest way to do something market-facing to start the iterative process that so frequently typifies innovation. Be prepared to make quick decisions, but have the driver of the decision be in-market data, not conceptual analysis. In other words, go small and learn. Pitch (or even sell) your idea to colleagues. Open up a kiosk in a shopping mall for a week. Create a quick-and-dirty website describing your idea. Be prepared to make quick decisions.

ChittahChattah Quickies

  • Forrester’s 2010 Customer Experience Rankings [Customer Experience Matters] – # Retailers take 12 out of the top 20 spots. Most of the top rated companies on the list are retailers. Hotels also grabbed three of the top 20 spots. Interestingly, three financial services firms also cracked the top 20: credit unions, SunTrust Bank, and Vanguard.
    # Healthcare, Internet and TV services dominate the bottom. The bottom 11 companies on the list came from only four industries: five health insurance plans (United Healthcare, Medicaid, Anthem, and CIGNA), three ISPs (Charter Communications, Comcast, and Qwest), two TV service providers (Charter Communications and Comcast), and one credit card provider (HSBC).
  • Can Design Change Behavior? [Stanford School of Engineering] – Because behavior can be influenced—not just observed—it provides an important opportunity for tackling complex challenges such as sustainability. That opportunity is perhaps best addressed with design…With this outlook, Banerjee says he is excited to be one of the principal investigators in a new project funded by the U.S. Department of Energy in which he is working with other Stanford professors who have expertise in behavioral sciences, communications, human-computer interaction, and behavioral economics. The team aims to create interventions that influence behavior to bring about significant reductions in energy use. But what designers understand well is that people are “predictably irrational” and influenced by emotional as well as rational criteria, Banerjee says.
  • The Art of Asking the Question [UIE Brain Sparks] – Steve Portigal will show your team the art of asking the question. You might visit the user in their office or home, have them come to you for a usability test, or even have a chance encounter at a trade show or while waiting for an airplane. Do you know what to ask? Do you know what to listen for, to extract the critical detail of what they can tell you about your design?

    Steve will help you prepare your team for any opportunity, be it formal user research or less structured, ad-hoc research. He’ll also give you tips on how to work with your stakeholders and executives, who may also be meeting potential customers and users, so they know what to ask and how to listen—integrating their efforts into the research team. (Wouldn’t it be great if they understood why you’re doing what you’re doing?)

    Update: Use promotion code CHITTAHCHATTAH to get lifetime free access to the recording after the fact (normally a separate cost)

Reading Ahead: Research Findings

Reading ahead logo with space above

(Updated to include slideshow with synchronized audio track)

We’re very excited today to be posting our findings from the Reading Ahead research project.

Lots more in the deck below, but here’s the executive summary

  • Books are more than just pages with words and pictures; they are imbued with personal history, future aspirations, and signifiers of identity
  • The unabridged reading experience includes crucial events that take place before and after the elemental moments of eyes-looking-at-words
  • Digital reading privileges access to content while neglecting other essential aspects of this complete reading experience
  • There are opportunities to enhance digital reading by replicating, referencing, and replacing social (and other) aspects of traditional book reading

We sat down yesterday in the office and recorded ourselves delivering these findings, very much the way we would deliver them to one of our clients.

Usually, we deliver findings like these to a client team in a half day session, and there’s lots of dialogue, but we tried to keep it brief here to help you get through it. (The presentation lasts an hour and twenty minutes.)

It’s been a great project, and we’ve really appreciated hearing from people along the way. We welcome further comments and questions, and look forward to continuing the dialogue around this work.


Reading Ahead: Building models

Reading ahead logo with space above

We’ve been hard at work synthesizing the Reading Ahead data. There’s a great deal of writing involved in communicating the results, and sometimes it makes sense to develop a visual model that represents a key idea.

Here are several partial models evolving through paper and whiteboard sketches, and finally into digital form.

We’ll be finishing synthesis soon, and publishing our findings on Slideshare, with an audio commentary.

Stay tuned…




Reading Ahead: Managing recruiting

Reading ahead logo with space above

There’s always something new in every project. Often we encounter a bit of process that we may not know how to best manage it. So we’ll make our best plan and see what happens. We learn as we go and ultimately have a better way for dealing with it next time.

In a regular client project, we write a screener and work with a recruiting company who finds potential research participants, screens them, and schedules them. Every day they email us an updated spreadsheet (or as they call it “grid”) with responses to screener questions, scheduled times, locations, and contact info. It still ends up requiring a significant amount of project management effort on our end, because questions will arise, schedules will shift, people will cancel, client travel must be arranged, etc. etc.

For Reading Ahead, we did all of the recruiting ourselves. Although we’ve done this before, this may be the first time since the rise of social media: we put the word out on LinkedIn, Twitter, Facebook, email to friends, and here on All This ChittahChattah.

While Dan lead the effort, we both used our own networks, and so we got responses in a number of channels, sent to either or both of us, including:

  • @ replies on Twitter
  • direct messages on Twitter
  • Comments on Facebook posts
  • Messages on Facebook
  • Emails (directly to either of us, or forwarded from friends, and friends-of-friends

A private dialog on Facebook

Comments on a Facebook status update. Note that Dan is able to jump in and make contact directly

Direct Messages in Twitter

Some people were potential participants, some were referrers to other potential participants, and some were both. And given the range of platforms we were using, with their associated restrictions (and unclear social protocols), we had to scramble to figure out who could and should communicate with who to follow up and get to the point where we could see if the people in question were right for the study. We didn’t expect this to happen, and eventually Dan’s inbox and/or his Word document were no longer efficient, and as some participants were scheduled or in negotiation to be scheduled, he ended up with this schedule cum worksheet:


Being split across the two of us and these different media, eventually we were interacting with people for whom we had to check our notes to trace back how we had connected to them, which was great for our sample, since it meant we weren’t seeing a group of people we already knew.

It was further complicated when we had finished our fieldwork and wanted to go back to everyone who offered help close the loop with them, thanking them for help. Technically, and protocol-wise, it took some work (who are the people we need to follow up with? Who follows up with them? What media do they use), basically going through each instance one-by one.


We haven’t figured out what we’ll do next time; we won’t forget the challenges we’ve had but there’s just not time or need right now to plan for the future. If I had to guess, I’d imagine a Google Spreadsheet that includes where we got people from, who owns the contact, whether they are participant-candidate or referrer, etc.). Despite being very pessimistic about the demands of recruiting, we still underestimated the time and complexity required for this project.

Reading Ahead: Looking for the story

Reading ahead logo with space above

I started today by typing up all of the Post-it notes you saw in our recent blog post on Synthesis.

This activity created a 6-page Word document of bullet points.

The next part of the process is something I always find challenging: taking an incredibly detailed list of observations, particpant statements, hypotheses, and ideas; figuring out what the Big Ideas are (there’s a point in the process where many of them seem Big!), and putting those into a form that tells a cogent story.

First step: make a cup of tea.

Ok, then my next steps were:

  • Categorize all those bullet points
  • Synthesize those categories a bit further
  • Write down in as short a paragraph as possible what I would tell someone who asked me, “what did you find out?”

Then I went into PowerPoint, which is what we use when we present findings to our clients. I’ll continue bouncing back and forth between Word and PowerPoint; each piece of software supports a different way of thinking and writing.

I dropped my synthesized categories into a presentation file, sifted all of the bullet points from my Word doc into the new categories, and then started carving and shaping it all so that it started to follow the paragraph I had written. (I’m mixing cooking and sculpting metaphors here.)

I printed out the presentation draft, and laid it out so I could see the whole thing at once.


Steve came back from a meeting and I asked him to read over what I’d printed out. He started writing notes on my printouts, pulling out what he saw as the biggest of the Big Ideas.


We talked about what he’d written, which led to an energetic discussion in which we really started to breathe life into this. Tomorrow, I’ll start the day by iterating the presentation draft based on our conversation.

Reading Ahead: Analysis and Synthesis

Reading ahead logo with space above

Synthesizing field data into well-articulated, data-driven patterns, themes, and opportunities is a big part of our work, but it’s an aspect that generally has less visibility than the fieldwork.

An essential early step in the synthesis process involves going back over the fieldwork sessions. An hour or two-hour interview creates an incredible amount of information. By going back into a record of the interview, we make sure not to leave anything significant behind.

We go through and make notes on interview transcripts (done by an outside service), watch videos of the sessions, and look over photographs, sketches, maps, and participatory design pieces.

Annotated interview transcript

We made a bulletin board of the people we met, so they’re ever-present while we’re working.


Yesterday we came together to share the points we’d each pulled out. We present each interview, like a case study, to the team. Sometimes it’s just us, and sometimes our clients join us for part of this process.


While one of us presented, the other captured the essence onto Post-its. We had a lot of discussion and debate while we did this, pulling together multiple viewpoints.


When we were done presenting the interviews, the board looked like this:


Our next step is to take these notes and start grouping them. We’ll look at different ways the information can be organized, and from there, will start refining our work and writing it up clearly and succinctly into a report.


About Steve