00:00The Associated Press reporting that Clinton is in a commanding position.
00:03Donald Trump will be the 45th president.
00:08Nationally it shows Joe Biden leading President Donald Trump.
00:12The race between President Donald Trump and Joe Biden still too close to call.
00:16Our new poll with Ipsos has Kamala Harris right now back in front.
00:20The next president of the United States is Donald Trump.
00:24Three presidential elections in a row. The polls pointed one way and reality went the other.
00:29Many people boil it down to a simple theory. People lied.
00:32But is that really what's wrong with modern polling?
00:34To find out, Straight Arrow turned to a swing state pollster,
00:37one of the nation's leading survey researchers, and a veteran TV news producer.
00:42Polling is strategic intelligence. Are you getting good intelligence or bad intelligence?
00:47And that matters because if not, you might be surprised.
00:50That's why I think a lot of people thought the election was stolen.
00:53Mike Noble of Noble Predictive Insights has been polling in Arizona for more than a decade.
01:06He says the problem isn't that people are dishonest. For him, it comes down to who is still willing to
01:12talk.
01:12People that have like, let's say, a college education, more white collar workers, they're taking surveys like it's a hobby.
01:18They have more flexibility in their schedule. They can sit there and do a five, 10 minute interview.
01:22However, what you're missing is that people like more blue collar jobs or high school or less education,
01:27they got time to stop in the middle of the thing and do a survey.
01:31So basically response rate wise, you get a big imbalance there.
01:3530 years ago, about one in three Americans would answer a pollster's phone call.
01:40Now that number is fewer than one in 50.
01:43The polling industry says it's adjusted for that, but Noble says that's harder than it sounds.
01:47Making a poll is like a chef, a chef. You only see the final meal that comes out.
01:53But again, the ingredients used, how he prepares it, blah, blah, blah, blah, blah.
01:56That's how basically polling does. So if you follow this recipe, you should get this result.
02:02John Krosnick, a professor at Stanford University, has spent his career studying exactly how people answer survey questions.
02:09So when he went searching for evidence that voters were hiding their true opinions from pollsters, he kept coming up
02:15empty.
02:16Researchers who want to claim that social desirability is a big problem can point to those studies as long as
02:22they don't point to how big the effects are.
02:24The minute they start paying attention to the fact that the effects are pretty small, then it becomes less of
02:29a problem.
02:30When Krosnick and his team reviewed every study comparing what people say in surveys to what they do.
02:35What we discovered is that about 85% of people's self-reports in surveys are accurate.
02:42And that's often, that's quite a bit higher than many professionals guess before they know what that literature says.
02:49But what about more sensitive questions, where telling the truth might come back with a social cost?
02:53Krosnick set out to explore the merit behind an oft-touted theory on why people conceal their true views, racial
03:00attitudes.
03:00We found no more evidence of racial prejudice being expressed, for example, by white people against black people in the
03:09anonymous condition than in the oral condition.
03:13And so that's yet another piece of evidence, in my opinion, that really raises serious questions about the claim that
03:22you can't trust self-reports because people are lying.
03:25Despite it being a popular theory, there's weak evidence that people are lying to pollsters.
03:29That raises the harder question. If voters are mostly telling the truth, why do polls keep getting it wrong?
03:35It's the abandoning of random sampling that's the problem.
03:38The night before the 2016 election, aggregated polls gave Hillary Clinton a 75 to 80% chance of winning.
03:45Most of those polls were opt-in surveys made up of people who volunteer to take surveys online, sometimes for
03:51money.
03:52In the battleground states that decided the election that year, roughly 70 polls were conducted in the final week.
03:57The scene here is so different than it was a few hours ago.
04:01The random probability samples had an average error in predicting the Trump and Clinton share of the vote of less
04:07than one percentage point.
04:09Really accurate. Whereas the average error of those opt-in sample surveys was much higher and sometimes a great deal
04:17higher.
04:17We just didn't have good quality telescopes to be able to see the outcome of this election clearly in those
04:25places where it mattered.
04:27A few good telescopes were no match for dozens of blurry ones, and those blurry ones drove most of the
04:32headlines, which turned out to be wrong.
04:34Ben Bogardus spent years producing newscasts in Washington, Houston, and Jacksonville before becoming the chair of the journalism department at
04:41Quinnipiac University, a school that happens to host a major polling center.
04:46Bogardus says responsible newsrooms will evaluate the source.
04:49You want to go down and almost have a checklist. You know, who is this poll done by? Who paid
04:53for it? Who commissioned it? Is it an independent poll? Is it by, you know, a network? Or is it
04:57by one of the campaigns or an advocacy group?
05:00You know, what is the quality of the poll in terms of the internals? How much, how did they try
05:07to reach people? How much did they, you know, weight it? What's the sample size? What's the margin of error?
05:13And then you have to say, you know, what is the sort of the wording of the poll? Is it,
05:19you know, phrased in a non-biased, subjective way?
05:22Bogardus can rattle this off because of his many years of journalism experience, something increasingly missing in today's newsrooms.
05:28A lot of the people who do have experience, the lifers, are retiring or leaving the profession.
05:35And the younger journalists replacing them are facing even more pressures to be fast, be everywhere, and produce more stories
05:41with fewer people.
05:42So if a press release comes in with a great headline, it looks like a shocking news story based on
05:47the poll results, you know, you're going to be sort of interested in putting that in your newscast as something
05:54new, something different, something exciting.
05:56And on social media, where people are increasingly getting their news, the polling picture is even less in focus.
06:03You know, if someone doesn't click through the article to read the actual specifics of the poll, you know, they
06:07could get a sort of a lopsided view of what the poll actually says because, you know, you only have,
06:13you know, the two or two sentences on your social media post.
06:16The problem isn't just polling, it's the system around it.
06:20And despite high stakes, flubs, and diminishing public confidence in them, polls are still attention grabbers.
06:26Since the beginning of time, there's been this sort of problem in journalism where, you know, when you're covering political
06:31campaigns, it becomes more about the horse race.
06:33You know, it becomes about who's ahead, who's, you know, going down.
06:37It's not really about the issues that are out there.
06:40The audience wants a simple answer, the algorithm rewards a simple answer, and polls, stripped of context, provide the simplest
06:48answers out there.
06:49Now, polling isn't broken.
06:51John Krosnick's research shows that when the right methodology is used, polls are still remarkably accurate.
06:57Mike Noble has built a career on getting those methods right.
07:00And Ben Bogartis spent years in newsrooms that strive to report polling responsibly.
07:04When done correctly, polls are still the best tools we have to understand public opinion.
07:09That is, when you know how to interpret them.
07:12I don't care what you say about prediction markets, all this other stuff.
07:15It is still the best tool out there.
07:18Is it infallible perfect?
07:19No, because errors can happen in that process of a poll.
07:23However, it's still the best thing out there.
07:26The polls are coming.
07:28They always do.
07:28But the question isn't whether to pay attention.
07:30It's whether you're looking through the right telescope.
Comments