Meteorologists know that the 10-day weather forecast is so unreliable that being right on Day 10 falls into the category of a miracle. Public health officials can’t say with a straight face that today’s dietary recommendations have legs that will sustain into the next decade. It is said that economists have predicted five of the last two recessions – because the predictive accuracy of the dismal science is, well, dismal. Psychologists don’t really know who will or won’t turn violent. Educators are mindful that paying teachers for performance sounds a lot better than it is. And physicists appreciate that human understanding of the universe sits atop a wobbly foundation because there yet exists no theory of everythingto fit the cosmos solidly together.
In fact, experts in every field are aware of the Achilles heels that diminish the certainty of the craft they ply. But experts aver, nevertheless, typically keeping their self-doubts private just as leaders lead even when they don’t know which way to go. So what about pollsters? Uncharacteristically, today, with dark curiosity, we can witness among professionals charged with telling us what we think an unusually public struggle over the secret behind how we know.
I have written recently that the evidence Americans rely on to understand society’s predilections has been based on one kind of public opinion survey method that is under assault. Though mostly unspoken, the primary model for collecting data over the last 40 years – random selection (“probability samples”) of Americans interviewed by telephone – is known by virtually every survey research professional to be unsustainable because phone surveys get ever lower response rates at accelerating costs while attracting people that don’t necessarily represent the general public.
As acclaimed pollster and analyst Nate Silver put it: “there are fewer high-quality polls than there used to be [because]…the cost to commission one can run well into five figures.” Another pollster put it more starkly: “The polling industry is clearly at a crossroads. In 2016, or 2020, telephone surveys may no longer be the prevailing way to measure what the public thinks or how it intends to vote.”
In the last week of July, the big secret went viral – at least for the pros in that virtual loop. A salvo by one of the most prestigious research and journalism teams –The New York Times/CBS News poll- vaunted for uncovering opinions the old-fashioned way – caused shock and awe among leaders of the survey research club – The American Association of Public Opinion Research (AAPOR). (Full disclosure, this author is a member.) The Times/CBS organization announced results from political polling that relied on the opinions of Americans who simply volunteered to be respondents in a Web panel curated by Yougov.
What seemed to some insiders like the abandonment of quality by respected journalists and researchers resulted, about one week later, in a censorious opinion piece by Michael Link, the president of AAPOR. He wrote:
“This week, The New York Times and CBS News published a story using, in part, information from a non-probability, opt-in survey sparking concern among many in the polling community. In general, these methods have little grounding in theory and the results can vary widely based on the particular method used.” Link went on to say, “Using information from polls which are not conducted with scientific rigor in effect sets a new — lower — standard for the types of information that other news outlets may now seek to report.”
This official AAPOR viewpoint has created aftershocks among members who think that opt-in Web surveys may be the right direction for learning what the people think and that the association’s scolding commentary was like the whining of buggy whip makers after the Model T rolled out.
Though there were supporters of the association’s position on the AAPOR-net listserve (e.g., “Based on my own experience as a pollster since 1976, none of our studies which allowed self-selection even remotely resembled the target population at large.”), some of the angry posts from members who support opt-in surveys were like these:
The American Association of Public Opinion Research is a justly well-respected organization … That is why the organization’s dismissive official reaction is so disturbing.
Sadly, AAPOR seems to be turning itself into a lobbying group in support of a certain segment [telephone surveying] of the polling establishment.
Isn’t it time to get out of the public censuring business? Surely this has gone too far.
We need AAPOR to be the anchor that grounds our work, not drowns our profession.
The publicly discussed weaknesses of traditional survey research already are spawning innovations that refashion how we know. While the retooling is in progress, we get to witness this uncommon open battle to make one professional secret obsolete. In the coming years as data collection turns from only telephone to a combination of phone, opt-in Web surveys, paid panels, smartphone texting, listed emails, tweeting, U.S. Postal Service mail or drone interviewers (my idea), the secret of low response rates and the misalignment of people in survey samples with people in general will surely be replaced with new polling secrets, yet to be discovered.
