In Tuesday’s presidential election, a number of polling firms that conduct their surveys online had strong results. Some telephone polls also performed well. But others, especially those that called land lines only or took other methodological shortcuts, performed poorly, showing a more Republican-leaning electorate than actually turned out.
Our method of evaluating pollsters involves looking at all the polls that an organization conducted over the final three weeks of the campaign — rather than only its very last poll. For each of the two dozen polling firms that issued at least five surveys in the final three weeks of the campaign, counting both state and national polls, I have calculated an average error and an average statistical bias.
The bias calculation measures the direction, Republican or Democratic, that a firm’s polls tended to miss. The estimate of the average error in the firm’s polls measures how far off the polls were in either direction, on average.
Among the more prolific polling firms, the most accurate by the error measure was TIPP, which conducted a national tracking poll for Investors’ Business Daily. Relative to other national polls, its results seemed to be Democratic-leaning at the time they were published. However, it turned out that most polling firms underestimated Mr. Obama’s performance, so those with seemingly Democratic-leaning results were often closest to the final outcome.
Among telephone-based polling firms that conducted a significant number of state-by-state surveys, the best results came from CNN, Mellman and Grove Insight. The latter two of these firms conducted most of their polls on behalf of liberal-leaning organizations.
Several polling firms got notably poor results.
For the second consecutive election — counting 2010 — Rasmussen Reports polls had a statistical bias toward Republicans, overestimating Mr. Romney’s performance by about four percentage points, on average. Polls by American Research Group and Mason-Dixon also largely missed the mark.
One of the most well-known polling firms, Gallup, had among the worst results. Gallup has now had three poor elections in a row. In 2008, its polls overestimated Mr. Obama’s performance, while in 2010 they overestimated how well Congressional Republicans would do.
Some of the most accurate polling firms this year conducted their polls online. The final poll by Google Consumer Surveys had Mr. Obama ahead in the national popular vote by 2.3 percentage points — very close to his actual margin of 2.6 percentage points, as of Saturday morning. Ipsos, which conducted online polls for Reuters, and the Canadian online polling firm Angus Reid also fared well.
Looking more broadly across the 90 polling firms that conducted at least one likely-voter poll in the final three weeks of the campaign, polling firms that conducted their polls wholly or partially online outperformed others on average. Among the nine in that category, the average error in calling the election result was 2.1 percentage points. That compares with a 3.5-point error for polling firms that used live telephone interviewers and 5.0 points for “robopolls,” which conducted their surveys by automated script.
The traditional telephone polls had a slight Republican bias on the whole, while the robopolls often had a significant Republican bias. (Even the automated polling firm Public Policy Polling, which often polls for liberal and Democratic clients, projected results that were slightly more favorable for Mr. Romney than he actually achieved.) The online polls had little overall bias, however.
The difference between the performance of live telephone polls and the automated polls may partly reflect the fact that many of the live telephone polls call cellphones along with land lines, while few of the automated surveys do. (Legal restrictions prohibit automated calls to cellphones under many circumstances.)
Research by polling firms and academic groups suggests that polls that fail to call cellphones may underestimate the performance of Democratic candidates. The roughly one-third of Americans who rely exclusively on cellphones tend to be younger, more urban, less well-off financially and more likely to be black or Hispanic than the broader group of voters, all characteristics that correlate with Democratic voters.
The Techniques Behind the Most-Accurate Polls - Nate Silver via NYTimes.com
Gallup and Rasmussen’s polling should be heavily discounted in the future, unless they drastically revise their methods. Google and others that survey online came very close to accurately predicting the outcome of the elections, while those that rely on calling via phone — especially landlines — skew toward the old and white, meaning GOP.
