Hi Steve, thanks for summarizing, criticizing and posting this! I’ve got the ‘invitation’ yesterday as well and wondered if the survey will ever end Furthermore (but this might not have been under control of the students) the question haven’t been adapted to an international audience as well. The situation on Water, Brands, etc. seems to be a pretty different one here in Germany as we have (similarly to Beer) very regionalized markets for water and our legal rules on water from the tap are even stronger than water being sold via stores ….

It has just been recently when I was able to stop students from making appointments for interviews with design & business professionals. The overall task we’ve discussed was to get a better understanding what SME understand as Design Management in their context.

You bet: The first question on the questionaire was: “What is your understanding of Design Management in your context?” … I took me two more sessions until I gave my OK for making any appointment

Enjoy your weekend and keep us posted!

Thanks, Ralf, for putting some quality control on your students! Nicely done.

I had blogged a couple of months ago on bad survey design, following a particularly unpleasant personal incident I went through. Here is Steve Portigal who says (well, screams, going by the tone of his post ) Bad Survey Design. Please Stop! Steve on why badly designed survey is dangerous, says it sharp and succint - Garbage in, garbage out. It is true that just about anything can be proved using research, and bad questionnaire design is about the easiest way to begin that process

I agree with your critique on design research in education. I got my masters in industrial design and now work as a design researcher for a fitness equipment manufacturer. I learned everything I know, with regards to research, from my experience outside of school. Instead of receiving a report at the end of a project I am actually writing the protocols, heading up teams in the field, documenting, analyzing, and presenting the findings/concepts to the key decision makers. I see with my own eyes how someone interacts with a product or how they make decisions regarding their shopping process. There are simple techniques an instructor could demonstrate to help students understand the difference between quantitative and qualitative and how to do each correctly. Design research done by industrial designers instead of or in conjunction with marketing departments is really an asset. It should be introduced to the students as a career option for them when they graduate. Some design students have a nack for the front end research and should be given the tools during school to help refine their skills in order to make themselves more marketable.

God thank god you wrote this. Now I can point to it and don’t have to flame people myself. Thank you Steve

Zac - you’re welcome!

Corin - Thanks for your comments. I co-teach a course in design research to ID undergrads at CCA. It’s an entire course devoted to trying, learning, and hopefully understanding why some things do and don’t work in terms of giving you insight about how someone ELSE might approach a problem, product, need, etc.

Hi, I am a student in FIT for packaging design. We’re working on a project that’s designed to be marketed towards teens. I did an extremely simple survey, and I couldn’t agree with your article more. I really don’t know how to make one and no one ever told us how to make one!! Guess this is why big and small firms spend big bucks hire people who actually studied and knows this stuff… >

100%. This is why product research should be done by Psych grads and not CAD monkeys. Great design doesn’t show; it observes.

Regardless, this example of a survey on water bottle usage is probably a waste. Without a million-subject sample size you’d have to target it to a demo/psychographic profile, which limits your survey to people you *think* use your product and misses any surprise audiences. Faster and more rewarding: Get out to the stores and follow the water bottles home. Of course, then kids would have to get off their butts, but by the time you’ve booted SPSS to filter your survey data, we’ve found 5% of your market you didn’t know existed.

Besides, only tools and undergrads take surveys.

Original post: Bad Survey Design. Please Stop! by at [Technorati] Tag results for survey design Blog tag: Survey design Technorati tag: Survey design Pages: Start Tag: survey+design

Amen!
I have worked with the best firms in the survey research industry. Garbage in, garbage out is the best way to describe the difference between a survey designed and conducted by professionals and amateurs. This applies to both quantitative and qualitative interviews. There’s exactly the same issue in focus groups. And the issue is definitely compounded when considering international markets, not only in terms of survey design, but also in terms of data collection method. For instance an online survey may be perfectly appropriate to reach a consumer sample in countries where Internet penetration is high (such as US, UK, Norway) but be completely inappropriate in China where a very small fraction of the population uses the Internet and where conducting face to face interviews is not only more appropriate but also highly cost effective.

Hi,
Im a student and this design survey is a mockery. The student that put this together must have put it together just to tick boxes and get a better grade.

You are all right, it is complete garbage. Im glad to say i wouldnt produce such a time wasting survey. But think its a shame it makes students look so stupid. Im glad to say in my course we are taught about qualatitive and quantitative reasearch and that you cant have leading questions.

I think the first thing we were taught were not to have bias questions that lead people to put down the answer you want.

Ben - this is very reassuring. Thanks for providing (another) positive example to at least balance out the negative one I’ve shared!

So I agree with everything you’ve said. Not writing biased/leading questions is diffcult and necessary.

“Garbage in” can only be prevented by pointing out what is garbage AND an example of “clean.”

Is there some place that can provide a guide to put people on the correct path to research and surveys?

I could do a lot of good to this end in my corporation if I knew what to suggest INSTEAD of the tired and thoughtless websurveys we do now.

Sure, check out http://www.surveysystem.com/sdesign.htm

How To Design A Survey…

Last week I’ve received an email from a colleague of my design management network; maybe some of my readers got this as well Anyway, the email pointed to an online survey ‘designed’ and conducted by some students from a US Easter…

» Bad Survey Design. Please Stop!

reggy - interactive regex exploration Bad Survey Design. Please Stop! [IMG] Posted in Uncategorized | Comments Off

[…] (Via All This ChittahChattah) […]

Breadcrumb Navigation XT iBar - intelligent surface system The Psychology of Cyberspace Der Hype ums iPhone: Die Apple-Lüge Musicovery : interactive webRadio Schools of Thoughts 3 Collanos Workplace Bad Survey Design. Please Stop!

[…] Recently I wrote about a Bad Survey, exhorting design instructors to pay more attention to why and how surveys are being used by their students. I was reacting to a pretty bad example I had encountered and I ripped on it and offered some suggestions for improvement. […]

of Manila to see who were the people who actually made his product designs. Eyeopening and thoughtprovoking. The need for more rigor in design methodology and processes is made by Steve Portigal in this incisive look at bad survey design by students. He places the blame squarely on the professors who teach them. Whirlpool’s design chief Chuck Jones talks to FastCompany about the biggest challenge facing the mature design industry - how to account for the return on your investment in design? Here’s

Ethnography Done Right Last Thursday, Steve Portigal introduced a great cautionary note on the dangers of bad (read: quick and uninformed) survey design. Steve’s argument was that design educators do their students a disservice by implying that quantitative online surveying is easy and quick. That it’s important to know that a survey is not something that anyone can do without planning and forethought, an understanding

[…] Do read the whole post here, and enjoy. Comment on this post […]

[…] This article was linked to from Core77, thought it raised some good points. I’ll probably send out some surveys- to sort a few engineering and aesthetic issues, a la Druce’s trailer brochure. Anyone else looking at that approach? ‘When I am working on a problem I never think about beauty. I only think about how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong.’ R. Buckminster Fuller […]

[…] Your page is on StumbleUpon […]

Hi there, I’d just like to say that I agree and disagree with a couple points you made.

AGREE - Surveys need to be well written in a clear and concise manner.
AGREE - Good surveys take more than a day to write
AGREE - They shouldn’t be used as a substitute for actual research.

DISAGREE - “Survey design is a craft. If you haven’t studied it, you don’t know how to write a survey well, and the data you get is garbage.” Sure to extent writing a survey is a craft, but I don’t think that you should discourage others from doing so just because they haven’t studied the craft. I feel that to write a good survey you need to:

A: Understand the problem
B: Target the right people
C: Word the questions & format of the survey so that you extract the most useful information in the shortest amount of time.
I feel that if you can do the things listed above, write well and know how to use a spell checker there is no reason why someone can’t put together a really good survey.

DISAGREE - “They require math to plan (what does your sample size need to be to ensure that your results are valid?) and to analyze (regression analysis (or any other buzzword) anyone?)”
What happens if you’re researching a product that is used by relatively few people of whom you know, you don’t have the time to put together a large list of people to survey, but know that essential information can be extracted from those few people? And what if you suck at math?
I surveyed 8 people for my major project last year and the information I got was invaluable to my project. I know this will be the case for many other up and coming ID students this year and in years to follow.
I feel this post is rather negative towards those who aren’t trained in the art of survey writing and may discourage some from using a highly effective means of research.

Have a great day.

Ben - thanks for a thoughtful response. I don’t believe I am discouraging people from creating surveys, but rather encouraging them to learn how to do it well; in a way that will be the most effective.

Many people seem to think that it’s trivial to bust out a survey that will work. That’s very naive, and that’s bad research. Denying that it’s a complex process that requires expertise simply because that fact isn’t encouraging is head-in-the-sand behavior.

It is difficult. It DOES take know-how. Many things in life take study, practice, training, and effort.

I can pick up a pen in my hand and put it on paper. That doesn’t mean I can draw well. Or even draw “good enough.” I can operate a motor vehicle. That doesn’t mean I can be a long-haul trucker, or even operate a semi.

Further, claiming, as you do, that a naively-constructed survey produced good data is a fallacy. Your sample was 8? That’s not a survey. If you showed bar charts and graphs with percentages and breakdowns, you were engaged in fantasy, and something that many academics would question ethically.

There is a great deal of math (and other principles) behind creating and fielding and analyzing surveys. If you choose to ignore it, that’s your deal and best of luck.

[…] This comment in the bad survey design thread got me thinking further about where/when/what to do with surveys. It’s not my primary tool so some of these reflections take me a little longer than someone who makes their living as a quantitative researcher, for example. […]

[…] We need more flexibility that doesn’t cost anything. Yes, I mean free. I don’t have funds to finances surveys. I want to use the existing free tools of web 2.0. I also want to avoid Bad Survey Design. I took several quizzes, polls and surveys this morning while playing and I believe 90% of what I found was garbage. Both garbage in and garbage out. Fortunately there are good guides to survey design like the one posted by Creative Research Systems. […]

[…] הנה המקור. […]

[…] Garbage in, garbage out. [Link] […]