This morning’s Guardian headline was what Bill Deedes would call “a bit of a marmalade-dropper”:
One in 40 Iraqis ‘killed since invasion’
US and Britain reject journal’s finding that death toll has topped 650,000.
650,000 dead since the invasion? Say it ain’t so!
And of course, there has been no shortage of people who are more than willing to say precisely that, including the Guardian’s own Michael White. But before dismissing this study too hastily, it is worth reading the statistical analysis by Daniel Davies, also on the Guardian website, who argues that The numbers do add up.
First up, the headline figure of 650,000. Davies expresses some regret that this “point estimate of the number of excess Iraqi deaths” has been given so much prominence:
Point estimates are almost never the important results of statistical studies and I wish the statistics profession would stop printing them as headlines.
Such estimates “are only interesting in so far as they demonstrate or dramatise the answer” to the real question posed by the survey, namely:
As a result of the invasion, have things got better or worse in Iraq? And if they have got worse, have they got a little bit worse or a lot worse?
Despite attempts to disparage the survey as only examining “small samples extrapolated to the whole country”, the sample was in fact a large one: “12,801 individuals in 1,849 households, in 47 geographical locations”. This is comparable with political polls, which typically survey 2,000 people and achieve a margin of error of +/- 3%.
But even if we just look at that sample and make no attempt to extrapolate, the results are still “shocking”:
In the 18 months before the invasion, the sample reported 82 deaths, two of them from violence. In the 39 months since the invasion, the sample households had seen 547 deaths, 300 of them from violence. The death rate expressed as deaths per 1,000 per year had gone up from 5.5 to 13.3.
It’s worth reading over that a couple of times – and perhaps imagining those figures as applied to the 2,000 households in the part of town where you live; imagining a situation in which the overall death rate in your town has more than doubled, with the death rate from violence increasing seventy-fold.
(Incidentally, some might wonder if Iraqis hostile to the invasion might be “lying and systematically exaggerating the number of deaths”. But Davies points out in a footnote that “death certificates were checked and found in 92% of cases”.)
As Davies concludes, the answer to the qualitative question posed by the survey is clear: “things have got worse, and they have got a lot worse, not a little bit worse”. Even if one’s hunch – or hope – remains that 650,000 is an overestimate, it is clear that the official figure of 60,000 is almost certainly an even wilder underestimate:
If you go out and ask 12,000 people whether a family member has died and get reports of 300 deaths from violence, then that is not consistent with there being only 60,000 deaths from violence in a country of 26 million. It is not even nearly consistent.
One of the most illuminating points made by Davies is one of general application to any discussion of statistics:
How Would One Get This Sample, If The Facts Were Not This Way?
And in this case, only one answer is really possible: that the study was fraudulent:
If a Mori poll puts the Labour party on 40% support, then we know that there is some inaccuracy in the poll, but we also know that there is basically zero chance that the true level of support is 2% or 96%, and for the Lancet survey to have delivered the results it did if the true body count is 60,000 would be about as improbable as this.
As Davies concludes:
Anyone who wants to dispute the important conclusion of the study has to be prepared to accuse the authors of fraud, and presumably to accept the legal consequences of doing so.
Of course, the study’s authors have, alas, saved their critics the trouble of doing this, by offering the easy target of “650,000 dead”, allowing people to attack that attention-grabbing headline while ignoring the human catastrophe that is still revealed by these figures.