A Real Life Parable about Data and Hearing What You Want to Hear Regardless of What Has Been Said

A school district contracted with a research firm to conduct a telephone survey of local residents to determine if there was enough support for a parcel tax measure to move forward with it. The research firm was paid over $18,000 to conduct the telephone survey over a five day period. A total of 400 surveys/interviews were conducted and factored into the results. Fifty-five percent (55%) of those surveyed said they would support the measure, which falls short of the two-thirds required for the measure to pass, so the school board chose to abandon the measure at this time.

The survey also revealed that "only 14 percent of those surveyed think the district is doing a good job of providing high quality education or preparing students for a job," and 60% of those surveyed believe that overall management of the district is poor.  Sixty-four percent (64%) believe that the district is doing a poor to only fair job of managing public funds.

Ouch.

Now, there are many things about this whole process that I could discuss, from the fact that $18,000 is an exorbitant fee to pay for a telephone survey of 400 residents (yes, many reputable research firms, including my own, would do an excellent job for much less) to the fact that the district had other no-cost and low-cost ways of getting pretty close to the same information, but I'm going to focus on the response to the survey results.

Just about anyone I have discussed this with says something like, "Wow. It's pretty clear that folks in that town think the school district is doing a lousy job. The public doesn't trust them with their money."

Interestingly, though, that's not what the school superintendent got out of those results. Here's what the local newspaper had to say about that: "She was interested to learn that, based on the survey, the community most valued tutoring for students, curriculum that uses science and technology, and more opportunities for students to take advanced classes."  And then the superintendent was quoted, "We need to continue to help kids that need extra help, continue to challenge kids that need more (rigor), and we need to do that with current technology."

Huh?

While all of that may be true, it seems to me that the real message to get is that the community doesn't trust the school district and thinks it's doing a lousy job.  That's what needs to be addressed.

We could debate the value of spending a lot of money on data gathering efforts.  As an evaluator, I'm a believer in investing in data collection to help you demonstrate the value of your programs and evaluate their effectiveness so you can improve them. The questions that comes up is always, "How much money is too much to spend for evaluation and data collection?"

But even that is not the moral to this story. 

The moral to this story is this:  If you're going to spend anything on conducting a survey, be willing to really listen and hear what people are saying.  If you're not going to learn from what has been said, even a dime is too much to pay for the information.

------------------------------

Click here for a free webinar on Tips for Conducting Focus Group Interviews.

Click here to access two free webinars on the Basics of Program Evaluation.