As Tuesday night got going, most people suspected that Republicans were headed for a good night. This, after all, was what the polls strongly suggested. Instead, they had a great night.
What is odd is that, while there was a cluster of conversations suggesting we might see Republican gains of around six-to-eight Senate seats, six-to-nine House seats and the loss of a few governorships -- and a secondary cluster analyzing why Republicans might be disappointed (mostly citing the possibility that polls could be skewed away from Democrats by demographic undersampling) -- there was no cluster of conversations suggesting that the public polls might understate Republican performance. As the New York Times Nate Cohnput it: Im not aware of any evidence that polls underestimate Republicans. Please show me the data.
What makes this even odder is that, at least with the benefit of hindsight, there actually was a lot of evidence out there, both that a Republican overperformance should occur and that it was occurring. For some reason, the bulk of pollsters and analysts simply werent picking up on it.
I dont have any firm answers as to why this would be the case, but before we can know why something happened, it useful to know exactly what did happen. The evidence that Republicans should have a massive win was well discussed, albeit only in certain quarters of the punditry. Back in February, I suggested that if the presidents job approval was at 44 percent among the electorate on Election Day (as it was, according to the exit polls), that the Democrats would tend to lose nine Senate seats. The piece was billed as an imprecise heuristic, so I dont want to oversell this. The point is, we had a guiding star in Obamas job approval, and it wasnt pointing toward a good result for Democrats. Political scientists like Alan Abramowitz and John Sides put out models with roughly the same results. At the same time, political science models of the House of Representatives favored GOP House gains somewhere in the mid-teens.
So we had a pretty solid basis for believing that Democrats performance would mirror the presidents job approval overall, and if there were going to be a poll error, it should probably be in the Republicans favor. So why didnt more polls show this? There were two separate reasons.
First, for much of the late summer and early fall, the polls actually were converging on the fundamentals, as Democrats remained tethered to the presidents approval rating, while Republicans gained ground. Mark Blumenthal expanded upon this phenomenon here. I suspect that high-quality private polling showed these trends continuing, before a final, 1980-style break at the end as undecided voters, who disapproved of the president, opted to vote Republican. This scenario is one that I outlined Thursday before Election Day:
The first scenario involves the working hypothesis Ive used for most of this cycle: Gravity wins out. In that scenario, Democrats are effectively capped by the presidents job approval, and undecided voters break heavily toward Republicans. Indeed, most of what we see right now is perfectly consistent with this theory. Democrats tend to run a few points ahead of the presidents projected job approval in their states: They are generally polling in the low 40s in the red states, in the mid-to-high-40s in the purple states, and in the low 50s in the bluer states.
Under this scenario, we should expect to see somethingakin to what weve seenover the past few months: A gradual improvement of the GOPs position in most races. The most prominent example of this is in New Hampshire, whereScott Brown has won over virtually every undecided voter, but you can also see agentler version of thisin the Virginia Senate race. This would be something like what occurred in 1980 and 1994, when we really were blindsided by a number of the races that broke for Republicans. If this happens, wed see double-digit gains for the GOP in the House and probably nine or 10 seats picked up in the Senate.
But some pollsters simply gave up on races too early to see the trend lines through -- these races gave us our biggest surprises. They committed the same error that Gallup committed back in the summer of 1948: They found a candidate ahead, concluded that he or she was likely to win, and stopped polling. In Vermont, the last public poll was completed in mid-October. It showed Peter Shumlin receiving the 47 percent that he eventually received. In Maryland, there were polls in early October that showed a substantial narrowing of the race, with Anthony Brown at the 48 percent that he eventually received. But there was only one subsequent public poll of the race.
Finally, the Virginia Senate race had been gradually converging on the fundamentals for the entire cycle, with almost all of the undecided voters headed Ed Gillespies way. Christopher Newport University, which had consistently been Mark Warners best poll, showed the senator up 22 points (53-31) in July, 51-39 in late September, and then only seven points (51-44) in late October. Roanoke College narrowed from a 20-point Warner lead in mid-September to a 12-point Warner lead in late October. These polls were flashing a red warning sign, and if any other pollster had bothered to test the race in the 10 days before the election to catch the final break, we probably would have seen the true scope of the challenges facing Warner.
The second thing is more complex. Some pollsters really did catch the full extent of the trend. In individual races, these tended to be the local pollsters. In the Maryland governor race, the Baltimore Sun and Washington Post caught the narrowing, though those were surveys taken almost a month before Election Day and for some reason they did not retest the race. In Virginia, the college polls caught the narrowing.
In the last week of October, we started seeing more and more pollsters releasing what looked like outlying polls: Selzer in Iowa, the University of Arkansas in the Natural State, Charles Franklins Marquette poll in Wisconsin. All of these were high-quality local pollsters who were basically projecting that the election was doing what the fundamentals said it should do. They were finding a much stronger break toward Republicans than the national pollsters were reporting.
This created an odd situation that you wouldnt catch unless you looked very closely. In statistical terms, the polls for a given race should, in theory, fall along what we call a normal distribution. And in late September and early October, there appeared in many of these states (according to a standard statistical test for normality) a distribution that did not look normal, but was rather bimodal.
A few days before the election, RCP had our new elections analyst, David Byler (who has an extensive background in statistics) investigate five states, using the polling for October. He found that New Hampshire and Colorado were normally distributed (in fact, the polls there were pretty good at the end). Kansas was also normally distributed, but had a huge standard deviation, meaning that while a close race was the most likely outcome, a large win either way was very possible. North Carolina and Iowa, however, were not normally distributed. This suggested that different pollsters really were doing different things. Unfortunately, it was hard to know which cluster was correct. One cluster had more pollsters in it, but the other cluster was more consistent with what the fundamentals were predicting.
This also showed up in national House generic polling. If you look at the generic balloting, a group of pollsters showed a wave building. CBS News, for example, had Democrats up four in the summer, but Republicans up six in early October, and then up eight in late October. ABC/Washington Post had Democrats up two in the spring, Republicans up three in September, and up seven in October. The Associated Press/GfKs final poll had Republicans up eight points. However, other national pollsters -- NBC/WSJ, CNN, FOX, Pew Research -- showed a tie or Democrats actually up one point in their final polls.
Overall we had a large cluster showing a tie in the race, and another cluster showing a substantial Republican lead. The way it was distributed was not consistent with random chance for large portions of the fall.
The day before the election, we received 435 YouGov House polls, finding Republicans were favored to pick up seats. A Monte Carlo simulation revealed that they were consistent with Republican pickups in the mid-to-high single-digit range.
As I went through them, to update our race ratings, I noticed that they were fairly consistently to the left of what publicly available House polls -- which were typically conducted by local outfits this election -- were showing. In fact, on average, YouGov was four points (net) to the left of the local polls. If you moved all of the YouGov polls four points rightward, and re-ran the Monte Carlo simulation, it showed Republicans with a pickup range in the mid-to-high-teens overall.
On a whim (there really isnt much to do before the polls close), I calculated a quick overall generic ballot test from these 435 YouGov findings. It showed that YouGovs released House polls were roughly consistent overall with the large cluster of generic ballots we discussed earlier showing effectively a tied race.
However, when the YouGov House polls were shifted four points toward the Republicans, to line the results up with the publicly available individual House polls, the national ballot came out consistent with a five-point Republican lead, which was consistent with the smaller polling cluster showing the Republicans with sizable lead.
By this point, it had become apparent that different pollsters really were approaching the potential electorate in two different ways -- one of which saw small gains, and another of which saw a huge wave. It was also clear that the wave polls were more consistent with what the fundamentals suggested should occur. But again, which group was correct? It was impossible to say ahead of time. But the evidence was nevertheless there that Tuesdays outcome was a distinct possibility.
At the same time, the fact that we can figure this out with hindsight is useful. It is always nice to be able to resolve data. It is also useful, as the question then becomes: What were these pollsters who were showing tighter races doing differently from those showing a widening spread? Why were they the larger group? Why werent the fundamentals receiving more attention, along with the fact that there really was a cluster of polls converging upon with these fundamentals?
I dont have good answers. The fact that the local pollsters (Des Moines Register, Marquette Law, The Arkansas Poll) who have well-developed skills in their states saw this coming seems telling. Regardless, this seems to be the what; until we start to hear pollsters telling their own stories, we can only speculate idly on the why.
Source: http://www.realclearpolitics.com/articles/2014/11/07/examining_the_polls_and_the_election_results_.html
