Stupid Survey Award, 2006
...goes to the Port Authority of NY & NJ.
Now, stupid is a juvenile word, one that implies a certain lack of vocabulary on the part of the person using it. In this post, I'm using stupid to mean, "senseless waste of time and money, clearly demonstrating little thought and making it likely that people will make bad decisions."
I have little respect for much of what happens at the unaccountable Port Authority, so this is par for the course. Here's the deal:
Leaving JFK, the helpful parking attendant at the cash register handed me an envelope that says, "Airport Parking Concept Survey". Inside are 23 questions (including income, where were you before you left for this trip and how many people did you fly with). Only five of the questions had to do with the topic at hand, which was, [summarizing]: if we built a valet parking facility, would you use it?
Why so stupid? Why worth posting about? Because it commited several survey sins, all at once:
- self-selection. The only people who would bother to fill this out are the ones in favor. Why would anyone opposed bother?
- fake census. They don't run surveys all that often at JFK, so there's no way to know if a 1% (or a 10%) response rate is any good.
- too much data (part 1): by asking all sorts of irrelevant questions, they depress response rate.
- too much data (part 2): by collecting all sorts of data (probably represented to three decimal points in the summary) they make the survey look a lot more accurate than it is.
- will know it when I see it: the biggest mistake, of course, is that no one knows if they'll use something like this in two years... it's too abstract to commit to.
"Why," a friend asks, "is it a bad idea for them to ask for feedback?" My answer is that they're not going to use the feedback because they actually want it, but because they intend to use it to sell the idea to others. They'll pick the data they like, make it seem quite significant and accurate, and it include it in a report. It'll tell a story. Which is why they are wasting their money (and our time) with a survey that doesn't do what a survey ought to do.