What Good Survey Results Look Like

During the School District’s Bond effort they did a “survey” of residents in order to understand what they’d be willing to pay for a bond. I think the answer was about $60 million. From reports it seemed like many survey calls were to cell phones of prominent people… which runs counter to the randomized nature you’d want in a study.

Then came the outcome of the failed bond and the school district. The school board decided they needed to do a study to understand why people didn’t think what the school board thought they would think. They reached out to people by phone and that led them to incorporating some of those people into focus groups. The results of those focus groups will likely be disseminated in March.

The problem is that it’s not transparent. We don’t know what was asked of the citizens by phone. We don’t know how the focus groups were chosen. We don’t know how many people were included. We don’t know how the focus groups were operated. We don’t know what questions were asked of the focus groups. We don’t know what answers were provided.

Yet, I’d guess the “answers” provided by the focus groups will be presented as perfect information.

So, how should it be done? I’m sure there are many different answers. In the past, I have heralded Summit County’s Citizen survey (produced by a professor of a university) for its methodology to determine the “state” of Summit County. It was a random, professionally done study that incorporated over 2000 people in our community.

However, I would present Park City’s National Citizen Survey as the gold standard in the area from what I’ve seen`. Park City has been participating in the survey since 2011. The survey is centered on the livability of Park City. What is fabulous about the survey is that the appendices tell us exactly what questions were asked, how many people participated, and what their answers were.

For a data nut like me, it is perfect. You’ll be able to see stats like 65% of the surveyed population thinks they will be living here in five years. You’ll find out weird statistics like more than 30% of people surveyed spend over $2500 a month on HOA dues! It is interesting info. It tells you that the numbered of Parkites surveyed was 300 (about 4.5%). All in all, there were about 50 questions asked.

While you may disagree with the outcomes of the survey, its methods seem strong. I would like to see such detail around the specific questions asked, the responses revealed, and the numbers participating in every survey our city, council, or school district performs going forward. It’s truly the best way of using the information presented to help impact your opinion in an accurate way. Otherwise, you may hear on the radio, “Our focus group said that people want to build a million square foot field house on Kearns.” However the question asked, that led to that sound bite, might have been, “Do you want to cancel all athletics or build a million square foot field house on Kearns?” Without knowing the questions, answers, and participation… the data is basically useless.

Let’s hope that all of our political bodies start releasing survey information in the same way that Park City did with the National Citizen Survey.

For those who care, it portrays a much more accurate picture.

If you’d like to delve into the details of the National Citizen Survey, the best place to look is The Technical Appendices. It makes for some interesting reading.