Posts tagged “survey”

Snark about Bark

I’m rolling my eyes at this article about successful dog toy company, Bark. Specifically:

Bark’s design process begins with research. Packed in every BarkBox sent to 600,000 subscribers is a survey questioning dog parents about their beloved canine’s playing styles. The design team also gets anecdotal data from their Ohio-based customer service team who chat with with BarkBox loyalists about how toys were received. Based on user insights, Jensen and his team creates goofy toys that heighten a dog’s natural play style—chewing, fetching, or even destruction play.

“We know what a golden retriever in Kansas will like compared to a chihuahua in Seattle,” claims Jensen.

Some of what is grating is simply due to sloppy writing, but I am bothered by the hollow virtue-signalling around user research. Their methods are surveys – sent only to people who have purchased their product, and customer service reps – who are probably doing more troubleshooting than chatting with loyalists. The quote, then, implies a Big Data-style sense of insight across geography and breed, which is just untrue. They haven’t met any pet owners and they haven’t met with any pets!

I realize it’s expedient for PR to anchor your genius in customer-centricity, and I guess that’s a win. The founders are industrial designers and there would once have been a day when their innate brilliance would have been sufficient. But really, don’t cloak yourself in shallow methodologies and then claim you are doing everything based on research!

Customer Satisfaction at the Push of a Button


Image from HappyOrNot.com homepage

The New Yorker profiles HappyOrNot, the company that makes those kiosks to measure “satisfaction” (you may have seen them in airports or malls, they have a few colored buttons with different happy/unhappy faces). The article explains the history of the company and gives examples of organizations who are using the technology, what kind of data they are getting, and how they are acting on it.

On a large monitor on a wall of Javaid’s office, Jochym showed us several ways she’d devised to represent HappyOrNot data graphically, for presentation to other members of the organization. One was a stadium seating map on which the HappyOrNot terminals were identified by numbered, colored circles. “You can see all the terminals here, and you can move through the data hour by hour. The colors change as the ratings do.” The most consistently high-rated performer—even during the two most problematic periods for customer service, halftime and the fourth quarter—was a new vender, which, unlike most other venders, used the same, experienced work crew at every game.

Survey or Proselytizing?

This religious pamphlet appeared on our doorstep, asking a difficult question – Can the dead really live again? – and giving the prospective convert three choice

  • yes?
  • no?
  • maybe?

I couldn’t help but think of a survey that would ask you to force your thoughts about a complex issue into some easily-summable categories. Sadly, the rest of the pamphlet did not include any skip logic, where religious content was presented differently, depending on how one responded to this provocative lede.

thedead

Innovation commoditization reaches a new low

Earlier this week I stayed at a Marriott hotel. When I checked out, they were unable to get me a bill. My room service from 2 hours earlier was not in the computer. The clerk tried to raise someone on the walkie-talkie but it was to no avail. They offered to email it to me, but 36 hours later as I prepared to submit my travel invoice to my client, I still didn’t have the bill. I explored the website, dealt with several different types of support, and it still took another 12 hours to get the bill!

Today comes the inevitable customer-satisfaction survey. With the audacious subject line Help us innovate your experience at Marriott hotels.

innovate

Besides the horribly ugly phrasing (“innovate your experience”?) how hard must they be kidding here?

Someone has hypothesized that escalating the language of the invite they can increase their response rate, but outright lying is really not the way to start the dialog.

Customer satisfaction surveys are not a way to innovate. Sure, it’s possible that this type of tool could uncover unmet needs, but those are going to be the needs that they already know about, right? Honestly, when have you ever taken a corporate customer satisfaction survey that has done anything but treat you like an idiot? This sort of tool is only used for ass-covering, at best, and at worst for one group to preempt any negative feedback that might go to another group that oversees or funds them.

The word innovation has become a meaningless catch-all for any sort of improvement and here Marriott stoops even lower, using it as a proxy for any sort of customer interaction, despite the low likelihood of any change or improvement resulting.

Don’t Bother, Braun

Today, I am proud to carry on the lively tradition of eviscerating… I mean, learning valuable lessons from other folk’s attempts at research. I will be examining a Braun-fielded poll that appeared on my Facebook page. (Recent notable additions to the oeuvre include Jared Spool’s 19 Lessons from United Airlines on How To Build a Crappy Survey and Steve’s imagined reaction to a Netflix survey, Effective Concept Testing: Getting the Answers You Want to Hear!)

OK, here it is:

I have a few questions.

1) Who? The pollsters don’t seem to care that I am neither a fan nor a consumer of Braun shavers specifically, or of electric shavers in general. They’ve got the audience all wrong. To respond would merely be to sabotoge their data-set in response to the absurdity. Which of course I wouldn’t dream of doing!

2) What is the purpose of this (Part I)? What is the marketing or social media team going to do with this information? What is the business question behind this? Knowing, as they must, that the data will be terribly corrupted, they can’t possibly believe that they’re actually getting useful information. So maybe it’s just one of these crazy social media ploys that appears to be important research but is really a bit of marketing designed at the level of a made-you-look joke?

3) What is the purpose of this (Part II)? If they just want me to look, then what did they want me to think upon having seen this survey/ad/poll? Is is supposed to intrigue me into thinking, “GOSH now I do wonder how new and different Braun shavers actually are! Let me look into that and then get back to them on this relevant question.” (If so, where’s the link?) Or is it, “Wow – Braun makes electric shavers!” Or is it merely an unconscious, Pavlovian Braaaaauuuuunn they’re trying to get? “Oh yeah, Braun is a brand. I need a shave.” Or do they really just want me to answer their ridiculous question?

4) Can it make sense, please? Don’t ask me to compare Braun, a brand responsible for a wide variety of consumer products, to the more specific but still questionable category of “other electric shavers.” I can’t compare things that are not comparable.

5) “New and Different?” Really? New and different are not necessarily positives, especially as those attributes relate to whirring blades that come into contact with your body parts. Is this the most important consumer response that the marketing team is really hoping to understand, if, in fact, they are hoping to understand anything at all?

6) Wait… anonymous or not? The question mark there, which (I know, I know) is a what-does-this-mean question mark probably linking to a privacy policy still reads like a sleazy wink. Fingers crossed! Your response may or may not be kept anonymous.

Even though I’ve no doubt that this is an insignifiant throw-away in the overall universe of Braun marketing, it definitely made an impression. If you’re going to bother to ask people questions, know who you’re asking and make it seem, even just a little bit, like the whole exercise matters to you.

We’ve learned quite a bit from other people’s surveys many times before:
Bad Survey Design. Please Stop!
Son of Survey
Son of Survey Madness
Thank You For Voting
The Space Between Yes and No
Does Calling it a Report Card Make it Not a Survey?
Survey Revenge

ChittahChattah Quickies

  • [from steve_portigal] 19 Lessons from United Airlines on How To Build A Crappy Survey [UIE Brain Sparks] – [Jared's detailed deconstruction of a badly written and entirely inappropriate survey – on board a United flight before he can get to the WiFi login screen – let alone find out if there's even a charge for the onboard WiFi – reveals the tragic limitations of badly written surveys and puts the lie to people who shrug off bad questions with "Well, at least you learn *something*". Even more this blog post reveals the emotional and intellectual state of someone who is taking a survey; the external orientation most surveys lack or deny. Required reading.] My biggest worry is the next flight I’ll get on with wifi service will have the exact same survey. If that’s the case, I’ll probably answer all the questions differently, just to mess with their heads. After all, if they’re going to waste my time…

ChittahChattah Quickies

  • [from steve_portigal] newWitch Magazine – Cutting Edge Paganism – [Seen in a "magic" shop today during a post-fieldwork ramble] newWitch is a magazine dedicated to, featuring, and partially written by young or beginning Witches, Wiccans, Neo-Pagans, and other earth-based, ethnic, pre-Christian, shamanic, and magical practitioners. Everyone from Traditional Wiccans to potion-makers to Asatruar to eco-Pagans can find something in these pages. The one thing we all have in common is a willingness to look at the world, our magical and spiritual paths, and ourselves in new ways. We hope to reach not only those already involved in what we cover, but the curious and completely new as well.
  • [from steve_portigal] Can the Kindle and Its Ilk Ease Textbook Inflation? [Village Voice] – [Thanks @dastillman] Pace offered the Kindle to students with course materials already preloaded on the device. Students had the option to buy the Kindle (at a discounted price) at the end of the course. Student complaints ranged from difficulties in taking notes to clumsy navigation controls. The electronic annotation feature was especially “slow and cumbersome,” requiring students to manipulate a tiny button to underline passages and type notes on the Kindle’s ergonomically unfriendly keyboard. The photos, pictures, and diagrams in the e-textbook were all black and white and image quality was not quite as sharp as in print….Soares found time eaten away by technical issues. Kindle books have no page numbers, so it was a challenge to get all the students on the same page. “It’s one thing to read a mystery or novel on the Kindle, but the way you read a textbook is different. You are flipping back and forth while reading, and navigation was cumbersome, even with bookmarks.”
  • [from steve_portigal] Doomsday shelters making a comeback [USATODAY.com] – The Vivos network, which offers partial ownerships similar to a timeshare in underground shelter communities, is one of several ventures touting escape from a surface-level calamity. Vicino, who launched the Vivos project last December, says he seeks buyers willing to pay $50,000 for adults and $25,000 for children. The company is starting with a 13,000-square-foot refurbished underground shelter formerly operated by the U.S. government at an undisclosed location near Barstow, Calif., that will have room for 134 people. Vicino puts the average cost for a shelter at $10 million. Vivos plans for facilities as large as 100,000 square feet, says real estate broker Dan Hotes, who over the past four years has collaborated with Vicino on partial ownership of luxury homes and is now involved with Vivos. Catastrophe shelters today may appeal to those who seek to bring order to a world full of risk and uncertainty, says Alexander Riley, an associate professor of sociology at Bucknell University.
  • [from steve_portigal] Market researchers get new tool in iPad [USATODAY.com] – [No doubt getting people to participate in surveys is an exercise in persuasion or seduction, but if there's a cool factor, something seems wrong to me] The gadget is luring curious consumers who've never seen one to participate in research projects conducted at shopping malls, primarily because they just want to see how it works. At many of the centers response was so good that survey takers collected the required information in about three weeks instead of the four they'd anticipated. The iPad presented its own set of research challenges. Some overheated in direct sunlight and shut down. In one case, a consumer at a mall in Rhode Island was so enamored with the iPad, he grabbed it from the interviewer and ran off.

ChittahChattah Quickies

  • Geek Power: Steven Levy Revisits Tech Titans, Hackers, Idealists [Wired] – The real problem, Greenblatt says, is that business interests have intruded on a culture that was founded on the ideals of openness and creativity. In Greenblatt’s heyday, he and his friends shared code freely, devoting themselves purely to the goal of building better products. “There’s a dynamic now that says, let’s format our Web page so people have to push the button a lot so that they’ll see lots of ads,” Greenblatt says. “Basically, the people who win are those who manage to make things the most inconvenient for you.” [Strongly worded insight about the state of Internet business rings tragically true /SP]
  • Organizing Armageddon: What We Learned From the Haiti Earthquake [Wired] – One of the biggest ideas to hit the humanitarian community in the past decade is the notion of surveying the recipients of aid to see what they think. That’s very commercial ­ treating them more like clients than victims…After the Asian tsunami, the Fritz Institute conducted one of the first-ever surveys of aid recipients. Only 60 percent of families surveyed in India and Sri Lanka said they had received timely aid and were treated with dignity in the 60 days after the tidal wave hit. Almost everyone reported getting water within the first couple of days, but just 58 percent of Sri Lankans reported receiving shelter in a timely manner. In general, post-disaster studies tend to measure “throughput indicators” like how much food was distributed, or how much shelter got provided, instead of “output or outcome metrics” like lives saved or suffering alleviated. [A powerful reframe on saving lives, with more cultural shifts clearly needed. /SP]

Bart Imitates Life

I was quite amused to see two topics near to my heart appear on The Simpsons last weekend. This this episode, the Simpsons travel with Ned Flanders and other Springfieldians to Israel. Ned gets very fed up with Homer and explodes: “You come all the way to Jerusalem, the happiest place on earth, and all the photos in your camera are of funny soda pops!” Yes! My Museum of Foreign Groceries (including Israeli beverages)! Here’s Homer’s pictures:





The episode also hits on another favorite topic – bad surveys – when Marge is asked to evaluate her tour guide:

My Devo color is red

After embarking on a customer research process (see Focus grouping the future), Devo (yes, the band) is now running a color survey. Surveys? What’s not to love! While we encourage you to check it out (if for no other reason than the satisfying UI, one of the best we’ve ever seen in an online survey), we’ve picked a few choice questions as a teaser. My Devo color is red. What’s yours?



Also see some fave survey posts from the past

ChittahChattah Quickies

  • Seen Reading – a "literary voyeruism blog" set mostly (I believe) in Toronto – What is Seen Reading?

    1. I see you reading.
    2. I remember what page you’re on in the book.
    3. I head to the bookstore, and make a note of the text.
    4. I let my imagination rip.
    5. Readers become celebrities.
    6. People get giddy and buy more books.

    Why do you do this?
    Readers are cool. Authors work hard. Publishers take chances. And you all deserve to be seen!

    (Thanks Suzanne Long!)

  • Choose What You Read NY – Choose What You Read NY is a non profit organization that offers free books to New Yorkers, encouraging its residents to read more, giving them an alternative to the free papers that get tossed out and even the digi-trash that crowds our time. In doing so, we help to recycle used books that would have unfortunately been thrown away.

    You will find us near major subway stations on the first Tuesday of each month.The idea is that once someone is finished with a book, they either drop it off in one of our conveniently located drop boxes or back to us at a station. Unlike a library, there will be no due dates, penalties, fees or registrations. We only ask that you return it once you are done so that the same book can be enjoyed by another commuter.

  • What was the last book, magazine and newspaper you read on the subway? – 6000 people respond and the New York Times posts the results
  • How and what people read on the New York City subways – Plenty of detailed examples of people, their books, and their travels: "Reading on the subway is a New York ritual, for the masters of the intricately folded newspaper, as well as for teenage girls thumbing through magazines, aspiring actors memorizing lines, office workers devouring self-help inspiration, immigrants newly minted — or not — taking comfort in paragraphs in a familiar tongue. These days, among the tattered covers may be the occasional Kindle, but since most trains are still devoid of Internet access and cellphone reception, the subway ride remains a rare low-tech interlude in a city of inveterate multitasking workaholics. And so, we read.

    There are those whose commutes are carefully timed to the length of a Talk of the Town section of The New Yorker, those who methodically page their way through the classics, and those who always carry a second trash novel in case they unexpectedly make it to the end of the first on a glacial F train."

    (thanks Avi and Anne)

  • Lego grabs ahold of customers with both hands – From 2006, great Wired piece about Lego's approach to involving ardent fans/customers in developing future products.
  • Noting:books – the simple yet dynamic way to track your reading, from the dates you start and finish a book, to your thoughts along the way.
  • CourseSmart brings textbooks to the iPhone in PDF; major readability challenges ensue – “It’s not the first place to go to read your textbook,” Mr. Lyman said of the iPhone app. But he said that it could be helpful if “you’re standing outside of the classroom, the quiz is in 10 minutes, and you want to go back to that end-of-chapter summary that helped you understand the material.”
  • Nice profile of Lego’s business culture and the tension between growth and losing track of their legacy – But the story of Lego’s renaissance — and its current expansion into new segments like virtual reality and video games — isn’t just a toy story. It’s also a reminder of how even the best brands can lose their luster but bounce back with a change in strategy and occasionally painful adaptation.

ChittahChattah Quickies

  • Summer Reading Programs Gain Momentum for Students About to Enter College – Nationwide, hundreds of colleges and universities, large and small, public and private, assign first-year students a book to read over the summer, hoping to create a sense of community and engage students intellectually.

    While there are no reliable statistics on summer reading programs, a recent survey of more than 100 programs by a student researcher at Gustavus Adolphus College in St. Peter, Minn., found that most had started in the last four years, although a few go back decades.

    The range of books colleges use is enormous, covering fiction and nonfiction. Classics are largely absent, with most of the works chosen falling closer to Oprah than academic.

    Still, a certain canon of summer reading is emerging: books that are readable, short, engaging, cheap. Often, it helps if the book is a best seller dealing with some aspect of diversity, some multicultural encounter — and if the author will come to speak on campus.

  • Canada Reads — CBC Radio – Canada Reads celebrates five Canadian books for three months online, on the air and at public events. It all leads up to a week-long show hosted by Jian Ghomeshi. In this annual title fight, five celebrity panelists defend their favourite work of Canadian fiction. One by one, books are voted off the list, until one panelist triumphs with the book for Canada to read this year.
  • Beyond the Book – Beyond the Book: Mass Reading Events and Contemporary Cultures of Reading in the UK, USA and Canada is a 3-year interdisciplinary research project.

    Our main objectives are to determine why and how people come together to share reading through a comparative study of selected mass reading events.

    The mass reading event is a new, proliferating literary phenomenon. Events typically focus on a work of literary fiction and employ the mass media as a means of promoting participation in the themed activities and discussions that take place around the selected book. Beyond the Book uses research methodologies drawn from both the humanities and social sciences to investigate whether mass reading events attract new readers and marginalized communities. We also wish to determine whether this contemporary version of shared reading fosters new reading practices and even whether it is capable of initiating social change.

  • "ONE BOOK" READING PROMOTION PROJECTS (Center for the Book: Library of Congress) – "One Book" projects (community-wide reading programs), initiated by the Washington Center for the Book in 1998, are being introduced across the U.S.A. and around the world. Here's lengthy list of authors, communities, and dates.
  • The Big Read – The Big Read is an initiative of the National Endowment for the Arts designed to restore reading to the center of American culture. The NEA presents The Big Read in partnership with the Institute of Museum and Library Services and in cooperation with Arts Midwest. The Big Read brings together partners across the country to encourage reading for pleasure and enlightenment.

    The Big Read gives communities the opportunity to come together to read, discuss, and celebrate one of 30 selections from American and world literature. This initiative supports innovative reading programs in selected communities, providing engaging educational resources for discussing outstanding literature and conducting expansive outreach and publicity campaigns, and a Web site offering comprehensive information about the authors and their works.

  • Literary Reading in Dramatic Decline, According to National Endowment for the Arts Survey – (July 8, 2004) Literary reading is in dramatic decline with fewer than half of American adults now reading literature, according to a National Endowment for the Arts survey released today. Reading at Risk: A Survey of Literary Reading in America reports drops in all groups studied, with the steepest rate of decline – 28 percent – occurring in the youngest age groups. The study also documents an overall decline of 10 percentage points in literary readers from 1982 to 2002, representing a loss of 20 million potential readers. The rate of decline is increasing and, according to the survey, has nearly tripled in the last decade.
  • 15 Books That Have Stuck With You (yet another of those Facebook etc. "memes" that are more like chain letters than memes) – Pick 15 books that will always stick with you. Don't take more than 15 minutes. Tag 15 friends including me because I'm interested in seeing what books my friends choose.
  • My pictures from Belgium 2009 (345 of 'em!) – Here's the whole set on Flickr. I'll continue to blog highlights from the trip.
  • Google book project far from settled – As the deadline draws near for authors and publishers to opt out of a proposed legal settlement allowing Google Inc. to forge ahead with plans to scan millions of books, more opponents of the landmark deal are stepping forward, and the local literary world is growing more perplexed.

    "Smart people, major players that are sophisticated in the ways of publishing, are still at loggerheads," said Ted Weinstein, a San Francisco literary agent. He said they're not just arguing whether the deal is good or bad, "but still expressing disagreement about what exactly it will do. That's a problem."

Son of Survey Madness

We’ve posted any number of survey design critiques over the years, and here’s the latest, a close read of a question and the cues associated with different responses.

In response to the prompt How closely do you agree or disagree with this statement: “We saw business strengthening in the Spring, but it seems to be stagnant or falling off again. We thought we had seen the bottom, but now we are not sure.” we’re asked to move a slider between Agree Completely and Disagree Completely.
smiley
frowny

As we move the slider, the expression on the little green character changes, supposedly to provide an additional cue to ensure that our response is accurate.

But when we agree (a positive emotion), the guy is frowning. Because we are agreeing with a negative in which case we making a negative observation? So we feel negative? But the green dude isn’t mapping our feeling about the situation, he’s mapped to our response – our degree of agreement. We can feel positive about agreeing, even if the thing we agreeing about is negative (haven’t you ever exclaimed enthusiastically at someone that expresses a similar frustration to you? That’s being positive about a negative). The mapping here is wrong.

It’s further complicated by the indirectness of the prompt – that situation you are agreeing or disagreeing with – describing a situation going from positive to uncertain. How much do you agree or disagree with: something was positive but now it’s negative? In fact, besides being indirect and somewhat abstract, it’s also a compound question. You might agree that things were positive, or you might now. You might agree that things have gone downhill, or you might not. The question is asking you to agree ONLY to the cause where i) things were positive and ii) things have gone downhill. If you don’t agree with both of those, then what do you do? And since you can indicate the strength of agreement/disagreement, how will people interpret the question? I would suggest not very reliably!

Ironically, this is a survey aimed at providers of market research services, who should absolutely know better.

Series

About Steve