I was bemused to see that Feast of Love opened last weekend. Our last time at the movies was when The Simpsons Movie opened, and I participated in some intercept-market research at the theater.
Part of the lobby had been given over to these groovy looking kiosks, with a couple of guys in attendance, asking people who passed by if they would like to give their opinion about an upcoming movie. My age and gender qualified me to participate (woo hoo) and I went with one dude over to a kiosk. I was shown a couple of clips and responded to various questions, but the weirdness of it was that the test was designed have some screens operated by me, and some screens visible only to the interviewer. But they didn’t do it that way. So for various pieces where I was to click within multiple choices, the interviewer, who knew the testing software rather well, just whipped through the keypresses, bam->redraw, bam->redraw, quickly asking me the minimum to move to the next one. Okay, so he took care of it for me. But then this screen we were both looking at would display testing instructions such as ASK PARTICIPANT FOR OPINION OF BENEFIT OF DATE MOVIES. PROBE ON RELATIONSHIP, TIMING, COST. And of course, he wouldn’t even come close, he’d get the one line answer from me, and then he’d type in the quickest condensation of my answer: stay home.
After a minute or so, it became more about the two of us cooperating to use the software to get through test. I realized that my opinion didn’t matter; it’s hard to feel represented in a forced-choice discussion, and it’s unlikely one would continue to provide color when all that gets captured is minimal facts. Further, by exposing the instructions to me, his shortcuts became clear, and I ended up slightly co-opted into the testing process, giving up any sense of really delivering the full truth to this interviewer.
When we see “market research” number published to support some business decision, let’s keep in mind how poorly that data may have been collected (from the concept of how to collect that data, to the implementation of a data collection environment, to the staffing and execution of the data gathering). How reliable could any of this possibly be?