HUMOR (Lite): Why I Flunked the City’s Branding Survey
Have you heard about the City’s “branding” initiative? The Economic Development Authority is paying $50,000 for a video to give Falls Church an “image.” The first stage is citizen participation — you’re encouraged to answer a 10-minute online questionnaire to help marketing firm Smith Gifford figure out just what our brand image should be.
Now, I already have a reputation as a nay-sayer, so I probably won’t be believed when I say that I took the survey with an open mind. But I really, really did. Yet I still flunked the test.
WRONG, you say, if you’ve already taken the survey. There are no wrong answers — it’s all opinion, and it’s impossible to fail.
Nope, I failed it all right. I’m sure of it. Because, (a) I couldn’t understand the directions, and (b) while there might be no “wrong” answers, the survey is missing some “right” answers as well.
I felt bad about not being able to understand the directions until I read the name of the web host: SurveyMonkey.com. Shouldn’t that be a tip-off? I’m not sure whether Smith Gifford is using the free version of Survey Monkey or whether they sprang for the $19.95 per month “professional” version. But either way, there’s some monkey business going on.
Example: “Pick one of the following statements that is most true for you:
– I do not want the City of Falls Church to change at all in the long term . . .”
I guess that works — sounds a little restrictive though: “not change AT ALL?” But I’ll take it.
But wait! There’s more. The whole statement reads: “I do not want the City of Falls Church to change at all in the long term. I do not mind if my taxes go up because I do not want my city to change.”
Whoa — could there be an agenda here? I think I just picked a WRONG answer. In fact, you don’t have to read between the lines to realize that there’s only ONE “right” answer. Here are the three choices for City residents:
CHOICE #1: “I do not want the City of Falls Church to change at all in the long term. I do not mind if my taxes go up because I do not want my city to change.”
CHOICE #2: “In the long term, we must get new businesses to move to the City of Falls Church. All of the tax burden can not fall on residents alone and more businesses moving here would make living in the City of Falls Church more affordable.”
CHOICE #3: “It is fine with me if over time, due to high costs, the City of Falls Church becomes a part of Arlington or Fairfax County.”
So, as the man said, you have a choice: You can be a profligate, irresponsible retrograde living in the previous century but paying sky-high taxes. Or you can be an uncaring non-participant while the City is gobbled up by an evil neighbor. Or, only other choice, you can be a responsible pragmatist who understands that the City must adapt in ways to attract new businesses that will shoulder some of the tax burden.
When the survey is complete, wonder what percentage will have voted for #2? If it’s anything less than 95 percent, the author of the survey should be fired. Kind of like elections used to be in Mexico — the ruling party wasn’t happy with 95 percent of the vote; they wanted at least 98.
Here’s another one I failed:
“The people who do not live in the City of Falls Church think the residents of the city are: (Check all that apply.)” 17 adjectives follow, such as broad-minded, snobby, ethnically diverse, outspoken, liberal, independent, closed and small-minded, and so on. But I don’t really know what “people” outside the City think about City residents. I suspect that “people” are a pretty broad group, and that “people” might think a lot of different things. But my strongest suspicion is that “people” really don’t spend any time at all thinking about what adjectives apply to residents of the City. So I decided that none of the adjectives applied, and I didn’t check any of them.
WRONG! You must check at least one. The computer is emphatic on this point: no check-ee, no next page-e. The buck stops here. To get the monkey off my back I checked all of them.
Here’s a question that could generate an age-discrimination lawsuit from the AARP:
What is your current employment status?
– Working full time
– Working part time
– Full time homemaker
– College student
– Not working, but looking for a job
Again, until you check one of the above you can’t proceed. Well, so much for retirement. Ditto for students not yet in college.
Of course, many of the questions were straightforward, and I could answer them: gender, age, marital status, total household income . . . .
Wait — “total household income”: There are 5 options:
– under $35,000
– $35,000 – $50,000
– $50,000 – $75,000
– $75,000 – $100,000
– over $100,000
I don’t want to brag, but this IS Falls Church. Did the survey author price any real estate before choosing that range of total household income? They must have missed our article reporting that Falls Church tax returns rank 4th-highest in the nation for adjusted income. The average is well over $100,000, and that’s for “adjusted” income, not “total household income.” A higher top end would have made the answer to this question considerably more interesting. And how about asking: “If you had to buy the house you live in today, could you afford it?” That would be worth knowing.
But we digress. Let’s go back to where I said I couldn’t even understand the directions:
“Please rank the following from 1 to 10, where 1 is the most desirable and 10 is the least desirable:”
I wish Falls Church had . . .
– a central gathering place
– better walkable connections between places
– more browsy, fun shopping stores
– more night life
– a movie theater
– more restaurants
– more specialty learning places
– more parking
– more clothing stores
– shops for everyday needs
They all sound good to me, but the questionnaire specifically requires a “rank order.” I took that to mean I had to number them 1 through 10, so that’s what I did. I assigned “better walkable connections” as #1, “a central gathering place” #2, and so on. But later I wondered if they really want us to RATE each quality between 1 and 10, according to desirability. That’s what most surveys do, and it makes a lot more sense because a RANK order implies that #10 is not desirable when it fact it may be, but just not quite as desirable as #9.
Just to experiment, I took the survey a second time (Survey Monkey tries to prevent this but there are ways, there are ways) to see if I could assign a rank of 1 to each of the above. It turns out that I could. Am I the only one who is confused on this point?
The survey concludes with a request to “please invite your friends to take the survey as well. . . . The more people who fill it out, the more reliable the survey data will be.”
WRONG! Literary Digest Poll, 1936: Alf Landon easily beats Roosevelt, according to over 2 million respondents. And so we learned that the number of persons polled is not so important, but a representative sample is crucial. Its reputation ruined, the Literary Digest folded, only to be resurrected, apparently, as Survey Monkey.
By George Southern
May 23, 2009




The quality of survey results is always related to the crafting of the questions.Garbage in/garbage out
I haven’t taken the survey yet (from your description it doesn’t sound too good) – but I can say that, despite the name, SurveyMonkey is a pretty good tool. I don’t think it will help you put together a good survey but if you need to do a survey and need a tool I would recommend SurveyMonkey.
The survey in question was so bad in so many respects it was an embarassment. I can’t believe a professional firm like Smith Gifford conceived this. Maybe it was done at city hall? In any case, something this public needed significant oversight and clearly it did not have that. Did Mr. Shields or the Mayor take it before it went live? These are smart people and I think they would have identified problems with it. But then again, when you’re drinking the same Kool-Aid…
I took the survey! George’s write-up is spot on. I think no matter how many people take the survey they won’t get much value out of it. There might be a question or two in there that would be interesting to see how people respond, but I think a 1-2 question survey would have been a better way to get that info.
It’s too bad they didn’t produce a better survey. I think a lot of people in Falls Church like to share their opinions and a quality survey could have returned valuable data.
Forget the SurveyMonkey…bring on the CanvassingCanines! Okay, so I made that up. You’re read by the best and brightest in the City. Let’s make up our OWN questionnaire. Get suggestions from readers, and put together one that is doggone better! Bet, then, they’ll be no barking up any trees here!
Teddy,
What kinds of questions should the survey include?
Stan Fendley