Saturday, May 24, 2014

Why do ICM think that only 45% of Scotland's voting-age population is male?

This is something that I haven't really paid much attention to, so thanks very much to Calum Findlay for pointing it out.  In the datasets for their referendum polls, ICM are absolutely explicit about where they get the targets for their weightings from -

"Data were weighted to the profile of all Scottish adults aged 16+.  Data were weighted by sex, age, social grade and region. Targets for the weighted data were derived from Census 2011."

So what does the 2011 census say about the breakdown of Scotland's over-16 population by gender?

Female - 2,280,734 (52.08%)
Male - 2,098,338 (47.92%)

Essentially a 52-48 split in favour of women, then.  It's therefore a touch hard to understand why ICM weighted their sample for last weekend's referendum poll so that it looks like this -

Female - 54.74%
Male - 45.26%

As I always say in these situations, "pollsters may have their faults, but they're not blithering idiots".  ICM will have done this deliberately, and they'll have done it for a specific reason, but for the life of me I just cannot even begin to imagine what that reason could be.  As far I can recall, every other pollster is using something very close to the more obvious 52-48 weighting.  In case you're wondering, it's not because ICM think women are more likely to vote - these are the figures for the whole sample before turnout weighting was applied.  (And indeed they found that men are in fact slightly more likely to vote anyway.)

Although men are currently more pro-independence than women, adjusting the weightings so that men make up 2.66% more of the sample normally wouldn't make much difference.  But in the case of this particular poll, the roundings in the published figures worked against the Yes campaign, so simply getting the gender weightings right would be sufficient to reduce the rounded No vote after Don't Knows are excluded by 1%, and to increase the Yes vote by 1%.  In other words, the figures should be Yes 43%, No 57%.

And that's before we even take account of the fact that for a second poll in a row, ICM have far too few respondents who were born in Scotland, and far too many who were born in England.  Reweighting the sample to bring those numbers into line with the census data would reduce the No lead even further. (I make it Yes 44%, No 56%, although admittedly it was such a complicated calculation that my brain was turning to mush, so don't take those numbers as gospel!)

If I was going to hazard a guess, I would assume that ICM aren't even bothering to weight by country of birth, and although they've probably noticed that the figures don't tally up with reality, they're not addressing the problem due to 'institutional inertia'.  But the gender issue is much harder to explain, because getting the right number of men and women is one of the most basic tasks of any pollster in any poll.

Then of course there's the even bigger issue of the bizarre introductory question that ICM used, which may have unsettled people and influenced how they responded to the referendum question in the first place.  With almost every successive closer look, last weekend's poll comes across as even more of a shambles.

1 comment:

  1. The thing is, until their method change two polls ago they did have the correct gender balance. Between that, the failure to weight by country of birth, the strange introductory question and the volatility ICM have shown anyway, I'm not quite so sure that ICM will be the "gold standard" pollster this time around.

    ReplyDelete