Wednesday, 9 April 2014

Are political opinion polls biased?

Over the weekend, Winston Peters claimed that New Zealand first was doing "far better" among actual voters than recent polls suggest. In particular, he based this claim on his wealth of personal experience and decades in NZ politics, and said:

"The polls have been overly kind for National election after election, and against a real night result they've been far too high. For Labour they've been about right up and down but about right,'' Mr Peters told TV One's Q+A programme today.
"For the Greens, they've been excessively favourable to the Greens, always way above what they find they get. And we're always below what we really get, so your so-called 7 per cent polls are nonsense. We're doing far better than that.''
So the statement being tested is:
- Polls overestimate the party vote for National (NAT)
- Polls are roughly accurate for the party vote for Labour (LAB)
- Polls overestimate the party vote for the Greens (GRN)
- Polls underestimate the party vote for New Zealand First (NZF)

A paper written by Rob Salmond in 2004 determined that some individual political polls in NZ do have biases (the draft paper has a big DO NOT CITE OR DISTRIBUTE WITHOUT PERMISSION at the top so I won't go into it too much, but it's floating around on the internet) and that in particular Colmar Brunton tends to overestimate the NAT vote in comparison to UMR and TNS. However, that's between polls - in general, how accurate are the polls?

A cursory glance at the tables on this Wikipedia page would suggest that political polls in New Zealand weren't very correct in 2011. They indicated who was going to win correctly but from 2009 onwards it was National's election to lose. Technically, the final results are in the margin of error of the polls but that's becuase the margin of error is usually quite large, making the polls quite imprecise (and thus achieving accuracy is not as impressive). It points towards systematic errors in the ways that the polls are conducted, and in this case we are interested in which direction those errors occur (I apologise for mixing scientific and statistical jargon; I know I'm at terrible person).

To investigate this, we should compare a number of polls to the final election result for each post-MMP election (i.e. since 1996) and how accurate it was for NAT, LAB, GRN, and NZF. I was just about to start doing this research properly when it turns out that Gavin White (UMR) has already done this in February 2014, comparing the final poll and the election result, and published his results on the SAYit Blog. Not much point in me doing it again. The conclusion? Between 1999 and 2011:

- 16 had National too high, while 3 had them too low. The most any company had underestimated National's vote by was 2%, while the most a company had overestimated National's vote by was 9%. One poll has had National's vote above their actual vote by more than the margin of error at three of the last five elections.
- 5 had Labour too high, while 5 had them too low.
- 9 had the Greens too high, while 3 had them too low. That overstates the case a little, because the most any poll has been out for the Greens is 3.4%.
- 1 had NZ First too high, and 9 had them too low. 
The biggest difference was in 2002, when one poll had them 6% too low - mostly the differences are within 2%.

One way of looking at this further is to take the average (mean) error for these four parties across the 19 final polls included in this dataset. That shows us that the average error is:
National: 2.7% too high
Labour: 0.7% too high
Greens: 1.0% too high
NZ First: 1.5% too low.

- If the total for Labour + Greens is within about 2% of the total for National and its allies (whichever of ACT, United Future and the Conservatives makes it into parliament), then it's actually pretty much a deadheat.
- If NZ First gets 4% in most of the mainstream polls, then they'll probably pass the 5% threshold on election day.

So the original statement being investigated:
- Polls overestimate NAT - True
- Polls are roughly accurate for LAB - True
- Polls overestimate GRN - True
- Polls underestimate NZF - True

Turns out Winston Peters was right on the money, and shows that his wealth of experience in NZ politics should at least make him quite good at iPredict.

No comments:

Post a Comment