Media polls are wrong — and it’s not just a math problem
November 7, 2022
The polling industry has been in turmoil since the 2016 Donald Trump upset of Hillary Clinton in the presidential race.
While certainly, 2016 wasn’t the first time that polling was wrong– see Bush-Gore 2000, Truman-Dewey 1948 — it also wasn’t the last time and even wasn’t the worst time pollsters were wrong.
Because while the poll numbers were bad in 2016, they were even worse in 2020.
In 2016, pollsters mostly got the results wrong, but were closer to the final numbers and at least within the margin of error after the Trump victory.
In 2016, pollsters were undone by confidence scores that made highly inflated claims that Hillary Clinton was a virtual lock to win the presidency.
On the basis of a 45 percent-to-42 percent lead over Trump in its own poll — and within the margin of error of most polls — Reuters/Ipsos said Clinton had the presidency won, the only thing left was to count the ballots.
“With hours to go before Americans vote, Democrat Hillary Clinton has about a 90 percent chance of defeating Republican Donald Trump in the race for the White House, according to the final Reuters/Ipsos States of the Nation project,” wrote Reuters.
Yet Trump found a way to win enough states to take the electoral majority.
In 2020, the pollsters made a different error: consistently and highly inflating the percentage by which Biden was leading Trump.
On October 5, 2020, one month before the election NBC News claimed in their poll that Biden had a 14-point lead over Trump.
On October 28, 2020, CNN claimed that Biden had a 12-point lead in a CNN national poll.
As late as November 1, 2020, NBC again claimed that Biden had a double-digit lead on Trump at 10 points.
In the end, Biden’s victory was 4.4 percent, about the typical margin of error for any credible poll.
Statistically speaking, in other words, at least some of the polls, should’ve shown Biden and Trump tied.
Or at least close.
And the problem of inaccurate polling is not just tied to presidential politics.
Just one week ago, amidst signs that the Democrats are going to go down to a historic defeat in the 2022 Congressional midterm elections, a Politico/Morning Consult survey found Democrats with a five percentage point lead on a generic Congressional preference ballot.
This even as reliably liberal New York and Oregon face the likelihood of huge GOP pickups in Congressional elections there.
But inaccurate surveys regarding Congressional midterm elections are nothing new.
In fact, Congressional preference surveys are notorious for undercounting Republicans.
Jim Ellis, a private pollster at Ellis Insights in Washington D.C., said that Congressional preference surveys have traditionally undercounted Republicans for years.
“You just know that if the GOP is ‘even’ in a generic Congressional ballot preference, it means they are winning,” Ellis said.
So why is public polling so bad?
In an interview in September, pollster Dick Morris said that part of the problem is political bias, and part of the problem is mathematical and structural.
Pollsters are doing these polls online which brings in a lot of tech-oriented people who have a bias to vote Democrat, said Morris.
“Also, pollsters are providing financial incentives, which gives people rewards like the 1,000th person to participate wins a free trip to Hawaii,” he said, talking about the structural and mathematical bias of today’s polls.
But it’s the political bias that’s the most dangerous, according to Conor Maguire of award-winning pollster WPA Intelligence.
“Sometimes, some of these polls are really built with the [ideological] end goal in mind,” said WPA’s Maguire.
The polls circulated by media outlets and newspapers are as much about clickbait as they are selling ideology that subscribers and viewers want to believe.
WPA, on the other hand, is a private polling firm that has no incentive to give their clients anything other than the truth, said McGuire.
Keith Naughton, a principal with Silent Majority Strategies, who writes extensively about polling said that there are three main components contributing to the problem with polls.
“Number one, it’s harder to reach people. So you’re getting a lot more refusals. You have to contact more people in order to get an adequate sample,” Naughton said in an interview.
What people don’t realize about that is that reaching more people costs more money to do a survey which puts pressure to cut the sample size, which makes the real margin of error larger, even if public pollsters don’t want to admit it.
In the old days the only way to get in touch with somebody was the landline and that single point of contact has turned into multiple points of contact, such as mobile phones, text messages and online surveys each of which is hit and miss.
Another problem is that the academics in polling refuse to acknowledge and be transparent about the fact that they just aren’t as accurate as they used to be, said Naughton.
“It’s inexcusable and they need to be more upfront and more transparent about their own problems. They’ve really been trying to present themselves as more accurate than they are,” he said.
They are not acknowledging their difficulties in building an accurate sample and representing what real voting will look like.
And there’s one thing for sure in life: academics are often wrong but they are never in doubt.
According to Norton that refusal to look at the sampling problem means that if their sample is off by a little, their results can be off by a lot.
Some of the problems are honest mistakes, but some of them are deliberate attempts to cook the books by fudging the sample.
“Honestly, I think some of the mainstream media, like CNN, which has terrible polling, want to give their viewers what the viewers want to read or hear or see, so that they’re less interested in, and less motivated to provide accurate polling when it gives people bad news,” Naughton said.
Distrust of the media and of the government is also fueling an increased number of refusals by Republicans and conservatives to participate in polling.
“People don’t want to tell pollsters what they really think because the media has a very negative view of the Republican Party,” he said.
That just adds to the difficulty in creating an accurate sample for pollsters.
But there might be another simpler explanation too.
Society is changing quite radically. It’s not the stable, peaceful world that we knew 30 years ago.
The inability of pollsters to predict behavior likely reflects the instability that we currently have in our society.
Society is certainly a lot more fluid if a large segment of the population can’t even decide how many genders there are at any given moment and which gender they identify with, to just cite one example.
“Political parties are in the process of shifting. And whenever you’ve got political coalitions that are shifting, it may make accurate polling that much more difficult,” said Naughton, who added that it might take as many as a half-a-dozen election cycles for pollsters to understand what an accurate sample looks like now.
So don’t expect the polling problem to go away anytime soon.
Like a lot of things that have to do with Washington D.C., there are just too many incentives not to fix the problem.
And there are too many rewards for institutions like the establishment media that make money off the problem by selling ideology, and then claiming they had no idea their results were so wrong.