How Can You Be So Sure? Seeking the Gold Standard of Survey Accuracy

Humans are programmed to find solutions. But, even so, we’re not that great at it. Every day, from kids to public managers, people decide what works and what’s good. A small conclusion can affect your lunch while a big one might change the lives of 5 million immigrants.

Sure, we rely on guts and brains to make our choices. But if we really explore the foundations of our decisions, we might recognize that we often are shooting in the dark – or concluding wrongly that we can see clearly what the right actions should be. In the public sector knowing what the public thinks shines a beacon on what otherwise can be a tangle of possible paths to national well-being. Insight into citizen sentiment helps illuminate the direction that democratic institutions should follow.

Bill Clinton was pronounced the poster child for political decisions that were based on the winds of public opinion . Since then almost every president has suffered the same accusation. But knowing what the public thinks is neither a malicious activity nor an exercise reserved for national politics. Local councils and government staff, too, rely on survey results to set the course of communities. Elected officials use surveys to allocate resources, assess the success of policies, track performance or make long-range plans.

If surveys deliver public opinion used for setting and tracking community direction, how can we be confident they are accurate?

I’ve written elsewhere that the traditional touchstone of surveys is how well they predict or match voting. By this standard, the national and statewide pollsgenerally did well in the 2014 midterms, calling most races correctly and getting within about 4 percentage points of the actual vote outcomes. But in local government where surveys of residents have become a management best practice, there rarely exists a behavioral marker like voting results by which to judge the accuracy of a survey.

The value of an accurate survey is not in the opinions collected from the few hundred or so who respond. The value comes only from the extent to which those hundreds report perspectives or behaviors that reflect the thinking or activities of the rest of the community for which they are only proxies.

So how can we be so sure that when a survey shows that respondents think the water is bad, we have solid evidence that the population thinks the water is bad? Or if survey results show low ratings for clean streets that the target population of adults thinks the streets are not clean? Or if respondents think their economic future is bright are they really representing the same sanguine sentiment as their neighbors? The second part of this blog, appearing next month, will propose answers useful to any leader who relies on survey data to help make decisions.