Election polls especially suffer from two specific types of measurement error: (1) election salience among voters at the time of the poll and (2) strategic voting decisions at the time of the vote which are at odds with poll responses.
On point one, the research literature shows that the farther a poll is out from election-day, the more error prone it will be. Many explanations exist, but the most common one relates to diminished election salience among voters at the time of the poll. Put differently, at the early stages of the electoral cycle, people are not paying attention to the candidates and issues.
In this context, a disinterested voter population is also prone to the vagaries of events, e.g. party conventions, which have a momentary impact but diminish in effect, over time, as voters forget.
Pollsters can measure election saliency in a number of different ways. First, often pollsters employ a simple question, such as ‘are you paying attention to the election”. They also use candidate familiarity as a proxy for greater (or lesser) voter attention and election saliency. Whatever the measure though, voters typically only start paying particular attention close to election-day. Some studies show that this window varies from one day to several months before election-day depending on the specific circumstances.
In sum, polls are more variable when they are conducted at length from election-day. The average voter is worried about more relevant “bread and butter” and ‘quality of life” issues than politics and elections. And, as such, it is not until quite close to the election that voters begin to pay attention and hence their responses are more considered and polls more accurate.
To analyze error in election polling, I employ an often used and widely-accepted measure of poll accuracy or error, known as the Average Absolute Difference (AAD). The AAD is a simple difference measure which takes (1) the absolute difference between the actual results on election-day for a given candidate minus the vote share for that same candidate then (2) takes the average of each absolute candidate difference.
We find that, in two-way races, the AAD increases the more distant from election-day the poll is conducted (see table 1 below). Specifically, our analysis shows that the AAD one week out is 3.58%–approximately equivalent to the MOE for a “gold standard” survey sample of 1000 (3.1%). In contrast, the ADD is 9% a year out from the election. Two months before election-day – the approximate period when the CPD is reviewing polling – the AAD for two-way races is 5.5%. Again, comparing AAD and MOE gives a ‘rule of thumb” indication of the presence and effect of non-sampling error. At one week before the election, the AAD is minimal and estimates show little potential non-sampling error (3.58% versus 3.1%). However, at two months out, the AAD is larger than the MOE, suggesting problems with non-sampling error.
Table 1: Average Absolute Error in Two-Way Races
Time before election
|Average absolute difference||Average margin of error|
To answer this question, I use data sourced from public opinion research organizations. This includes data from 95 firms over 1,000 polls and approximately 2,500 observations. This includes polling firms such as CNN, USA Today, Ipsos, SurveyUSA, Field Poll, Gallup, Braun Research, Field Research Corp., Public Policy Polling, Quinnipiac, and state-level University and newspaper polls including, Brown University, Southeaster Louisiana University, Minnesota Public Radio, Los Angeles Times, Portland Tribune, Suffolk University, and others. These opinion research organization include most of the major media public opinion pollsters and include many of the same organizations relied upon by the Commission on Presidential Debates.