- Since early June, our Reuters/Ipsos horse race ballot question (Clinton versus Trump) has shown a larger spread (Clinton-Trump) than the average of the market. Specifically, over this time our poll has given Clinton, on average, a 10-point lead, while that of the market has been narrower at 5-points.
- So why the difference?
- We analyzed a number of factors including both sample profile as well as questionnaire design. Ultimately, we identified that our ballot question response options were different from other online pollsters in the market. (In this assessment, we limited our analysis to online pollsters given the self-administered nature of the questionnaire and associated mode effects.)
- What did we find?
- Specifically, we found that while Ipsos includes a combined “Neither/Other” option, the other online polling firms do one of two things: (1) provide a pure “Other” option (without the word “Neither”), or (2) do not provide any “Other” option at all, allowing only three response options (“Clinton,” “Trump,” and “Undecided” – or a variation on “Undecided”).
- Here it is important to point out that the “Neither/Other” option is a holdover from the days when we did live telephone interviewing. Such options were only voluntary and never explicitly stated.
- In order to isolate a potential question wording effect, we ran a three-ballot experiment which included (1) our original question wording—Ballot 1; (2) our question wording minus the “Neither” option—Ballot 2; and (3) the three-category question with no other option—Ballot 3. We randomly assigned respondents to one of the three ballots conditions with a sample of approximately 940 respondents per condition:
- Below we include the results of our experiment. We break the ballot conditions down by varying universe definitions: all adults, high turnout, medium turnout, and low turnout. We also include unweighted results for all adults as a benchmark.
- So what did we find? The results differ across the ballot conditions, and these differences are statistically significant. Specifically, the Reuters/Ipsos ballot approach (Ballot 1) maintains a consistently larger spread (Clinton-Trump) than do Ballots 2 or 3, independent of the universe definition. Additionally:
- On average, the “Neither/Other” option in Ballot 1 is five percentage points higher than that of Ballot 2.
- Both Ballots 2 and 3 behave in ways that intuitively make sense: the spread declines with decreasing turnout.
- The unweighted data for Ballots 1 and 2 look almost identical; it is only when the data is weighted that Ballot 1’s behavior is at variance with what one would otherwise expect. This strongly suggests that the ‘Neither’ option is pulling a subgroup with a distinct demographic profile.
- So what substantively is going on?
- In our view, the inclusion of the word “Neither” is capturing Soft Trump supporters who, if given such an option, prefer not to make a choice. Here it is important to note that the soft supporter phenomenon also affects Clinton, but to a much lesser degree.
- Most importantly, the demographic profile of the “Neither/Other” is distinct from that of the “Other” option. Indeed, respondents selecting “Neither/Other” are older and whiter than those selecting “Other,” and are also slightly more educated. This prima facie fits the profile of a Trump supporter who may lean Trump but is uncertain at this point of the election year.
Some Additional Points
- We ran the above experiment on our online Omnibus survey. This polling platform shares a similar sampling profile with the Reuters/Ipsos tracker.
- Additionally, we ran a parallel experiment on the Reuters/Ipsos tracker vehicle:
- Here we only experimentally tested Ballots 1 and 2. We achieved this by doubling our sample size so as not to impact our ongoing tracking.
- This additional experiment yielded similar results to those reported above, although the timing of this part of the experiment (mid-conventions) muddies the analysis a little bit, and our work is ongoing.
- From this experiment on the ‘live’ survey vehicle, newer data suggests it is possible that these “Neither” choosers are broadly soft supporters of either candidate, and that the candidates pull disproportionately from the pool of uncertain individuals depending on current political events or dynamics. While the experiment broadly suggests that this pool contains more soft Trump supporters, this is indicative only and data from the past few days (following end of GOP convention and first days of Democratic convention) finds more soft Clinton supporters in this pool. This makes intuitive sense (since during the convention, Clinton occupies more share of media voice) and will be an area for ongoing exploration and reporting by Ipsos and Reuters.
- Based on the results of these experiments, we have decided to change our traditional head-to-head ballot question: specifically, to drop “Neither” from our “Neither/Other” option. Why?
- In our opinion, this standardizes our approach bringing us into line with the market norm.
- Additionally, the inclusion of “Neither” adds a layer of ambiguity and endogeneity that is suboptimal in survey measurement. Uncertain voters should be screened out by our likely voter model and not by our ballot question.
Finally, as of July 26 our poll included this revision.