Figure 9.4
Number of homicides each year in England and Wales between 1998 and 2016, and 95% confidence intervals for the underlying ‘true’ homicide rate.4
There was an increase of 557 − 497 = 60 in the number of homicides between 2014–2015 and 2015–2016. It turns out that a 95% confidence interval around this observed change runs from −4 to +124, which (just) includes 0. Technically this means that we cannot conclude with 95% confidence that the underlying rate has changed, but since we are right on the margin it would be unreasonable to proclaim that there has been no change at all.
The confidence intervals around the homicide counts in Figure 9.4 are of a totally different nature to margins of error around, say, unemployment figures. The latter are an expression of our epistemic uncertainty about the actual number of people unemployed, while the intervals around homicide counts are not expressing uncertainty about the actual number of homicides—we assume these have been correctly counted—but the underlying risks in society. These two types of interval may look similar, and even use similar mathematics, but they have fundamentally different interpretations.
There has been some challenging material in this chapter, which is unsurprising as it has essentially laid out the whole formal foundation for statistical inference based on probability modelling. But the effort is worthwhile, as we can now use this structure to go beyond basic description and estimation of characteristics of the world, and start seeing how statistical modelling can help answer important questions about how the world actually works, and so provide a firm basis for scientific discoveries.
Summary
• Probability theory can be used to derive the sampling distribution of summary statistics, from which formulae for confidence intervals can be derived.
• A 95% confidence interval is the result of a procedure that, in 95% of cases in which its assumptions are correct, will contain the true parameter value. It cannot be claimed that a specific interval has 95% probability of containing the true value.
• The Central Limit Theorem implies that sample means and other summary statistics can be assumed to have a normal distribution for large samples.
• Margins of error usually do not incorporate systematic error due to non-random causes—external knowledge and judgement is required to assess these.
• Confidence intervals can be calculated even when we observe all the data, which then represent uncertainty about the parameters of an underlying metaphorical population.
CHAPTER 10
Answering Questions and Claiming Discoveries
Are more boys born than girls?
John Arbuthnot, a doctor who became physician to Queen Anne in 1705, set out to determine the answer to this question. He examined data on London baptisms for the 82 years between 1629 and 1710, and his results are shown in Figure 10.1 in terms of what is now known as the sex ratio, which is the number of boys born per 100 girls.
He found there had been more males than females baptized in every year, with an overall sex ratio of 107, varying between 101 and 116 over the period. But Arbuthnot wanted to claim a more general law, and so argued that if there were really no difference in the underlying rates of boys and girls being born, then each year there would be a 50:50 chance that more boys than girls were born, or more girls than boys, just like flipping a coin.
But to get an excess of boys in every year would then be like flipping a fair coin 82 times in a row, and getting heads every time. The probability of this happening is 1/282, which is a very small number indeed, with 24 zeros after the decimal place. If we observed this in a real experiment, we would confidently claim that the coin was not fair. Similarly Arbuthnot concluded that some force was at work that produced more boys, which he thought must be to counter the greater mortality of males: ‘To repair that Loss, provident Nature, by the Disposal of its wise Creator, brings forth more Males than Females; and that in almost a constant proportion.’1
Figure 10.1
The sex ratio (number of boys per 100 girls) for London baptisms between 1629 and 1710, published by John Arbuthnot in 1710. The solid line represents an equal number of boys and girls; the curve is fitted to the empirical data. In all years there were more baptized boys than girls.
Arbuthnot’s data have been subject to repeated analysis, and although there may be counting errors, and only Anglican baptisms were included, his basic finding still holds: the ‘natural’ sex ratio is now considered to be around 105, meaning 21 boys are born for every 20 girls. The title of the article he published uses his data as direct statistical evidence for the existence of supernatural intervention: ‘An Argument for Divine Providence, Taken from the Constant Regularity Observed in the Births of Both Sexes’. Whether or not this is a justified conclusion, and although he was unaware of it at the time, he had entered history by conducting the world’s first test of statistical significance.
We have reached perhaps the most important part of the problem-solving cycle, in which we seek answers to specific questions about how the world works. For example:
1. Do the daily number of homicides in the UK follow a Poisson distribution?
2. Has the unemployment rate in the UK changed in the last quarter?
3. Does taking statins reduce the risk of heart attacks and strokes in people like me?
4. Are mothers’ heights associated with their sons’ heights, once the fathers’ heights are taken into account?
5. Does the Higgs boson exist?
This list shows that very different kinds of question can be asked, ranging from the transient to the eternal:
1. Homicides and the Poisson distribution: a general rule that is not of great interest to the public, but helps to answer whether there has been a change in underlying rates.
2. Changing unemployment: a specific question concerning a particular time and place.
3. Statins: a scientific statement, but specific to a group.
4. Mothers’ heights: possibly of general scientific interest.
5. Higgs boson: could change the basic ideas of the physical laws of the universe.
We have data that can help us answer some of these questions, with which we have already done some exploratory plotting and drawn some informal conclusions about an appropriate statistical model. But we now come to a formal aspect of the Analysis part of the PPDAC cycle, generally known as hypothesis testing.
What Is a ‘Hypothesis’?
A hypothesis can be defined as a proposed explanation for a phenomenon. It is not the absolute truth, but a provisional, working assumption, perhaps best thought of as a potential suspect in a criminal case.
When discussing regression in Chapter 5, we saw the claim that
observation = deterministic model + residual error.
This represents the idea that statistical models are mathematical representations of what we observe, which combine a deterministic component with a ‘stochastic’ component, the latter representing unpredictability or random ‘error’, generally expressed in terms of a probability distribution. Within statistical science, a hypothesis is considered to be a particular assumption about one of these components of a statistical model, with the connotation of being provisional, rather than ‘the truth’.