Politics, Statistics

Annals of Bad Statistical Analysis

Nate Silver’s newly revamped site FiveThirtyEight has gotten its share of flak since launching, some of it more warranted than others. And I don’t mean to pile on; I think the site has great potential. However, this latest post by FiveThirtyEight’s senior political writer Harry Enten is so thoroughly flawed, it deserves special condemnation.

Enten seeks to show that “turnout isn’t nearly as important as D.C. wags make it out to be” as an explanation for the variability in election year results between midterm and presidential years (think the GOP wave of 2010 vs. 2008 and 2012). Enten argues that “what really mattered was that voters changed their minds about which party they wanted to vote for.”

Age differences as share of electorate in 2010 and 2012. Date from exit polls.

Age differences as share of electorate in 2010 and 2012. Date from exit polls.

To make that case Enten shows that within certain demographic groups, the voters who did turn out in 2010 simply favored Republicans more than they did in 2012. This would make sense if the voters were actually the same voters. But they weren’t. There was a fairly large differential in overall turnout: 41 percent of the voting eligible population in 2010 compared to 58 percent in 2012. So even if you could show that the characteristics of who turned out in 2010 vs. 2012 were identical, or at least identical in all the ways you are able to measure them, but simply held different views, this 17 point disparity poses a serious threat to inference about whether anybody actually “changed their minds.” It may well be that simply fewer pro-Obama voters across all demographics were motivated (or able) to make it to the polls in 2010 when he wasn’t on the ballot.

But there’s actually a more serious problem with Enten’s argument. The gist of what he contends is that yes, the electorate was older (and whiter) in 2010 than 2012, but not by enough to explain the difference in outcomes. The problem is that Enten treats age as two categories (under 30 and over 30) and in doing so washes out a ton of potentially critical variation. The problem isn’t just the difference between the under 30 vote, which varied between 12 and 19 percent of the electorate, but also that the 65 and over vote varied between 16 and 21 percent. And that over 65 vote tilted toward the GOP by 21 points in 2010. (Enten relies on CPS data, which like all retrospective voting reports contains problems with nonresponse bias and overreporting. Here I draw on the exit polls for 2010 and 2012).

Second, Enten doesn’t even consider other variables besides age and race. And some of these matter quite a bit given existing resource-based theories explaining voting participation. Wealthier people in general are more likely to vote, but particularly in lower salience elections like midterms.

Share of electorate by income level in 2010 and 2012. Data from exit polls.

Share of electorate by income level in 2010 and 2012. Data from exit polls.

And indeed this was true in 2010. The share of the electorate making under $50k was just 36 percent (compared to 41 percent in 2012). This share voted for Democrats by a 22 point margin in 2012 and by an 11 point margin in 2010. It wasn’t just the case that the midterm electorate was older and whiter, but also wealthier. All of which helps to explain a great deal about why the results were so different. And I would wager, given our polarized politics, that it also explains a great deal more about the outcome than theories about a large number of voters actually “changing their minds” about the parties.

A more formal test of whether attitudes or demographics explain differences in the voting outcomes in 2010 or 2012 requires a regression analysis examining the relationship between these and other factors at the individual level. I don’t bother going down that route because that’s not really my point. (Plus with so many fewer voting in 2010 altogether compared to 2012, I would have serious concerns about omitted variable bias related to other factors that explain turnout from one year to the next that you may not be able to control for with the exit poll questions alone).

My point is: I understand where Enten’s coming from, and a healthy skepticism about conventional wisdom is admirable. But his latest strikes me as emblematic of the site’s problems. There’s a little too much emphasis on using numbers to make a counter-intuitive claims, and not enough emphasis on actually using the numbers in a truly rigorous, systematic way. I have a lot of respect for what Nate Silver is up to, but that’s precisely why I expect more from his site.



Comments are closed.

Follow me on Twitter

%d bloggers like this: