Appreciating statistical variation to improve the interpretability of polls

The 2016 European Union (EU) referendum produced one of the biggest surprises of the 21^st century. Most polls leading up to the referendum suggested that the majority of UK would vote Remain in the EU. However, the referendum produced a different result with 51.9% voters wanting to Leave the EU. Since then, there have been chaotic scenes on whether and how Brexit would be enforced. The polling industry has also been under attack with debate surrounding whether polls are still useful for predicting how the population would vote on important issues such as Brexit.

What the mass media and the general public do not appreciate is that polls, whose results are taken as the overall view of the population, only survey one small part of a population that might vote differently from the sample. This introduces variation in the polling result which might introduce a situation where it does not favour one side over the other. Hypothesis testing and confidence intervals can be used to decide whether the public should care about a polling result and the range of referendum results that are possible from a poll. If communicated simply to politicians and the public, more informed decisions can be made on what, if any, one should do to influence people’s views towards voting Remain or Leave in the EU referendum.

How do opinion polls report variation in results?

Opinion polls are conducted on people drawn randomly from a population to gauge the population’s views of an issue. It is like tasting a small sample of a meal such as soup and using our thoughts of the sample to make a general judgement of the meal. In our case, we use statistics to extend the results of the sample to make conclusions about a population. The problem of this method is that people outside the sample may vote differently from those in it, causing population results to differ from a poll.

Hence, in statistics, it is important to account for the variation in polling results to capture the true value of the population. This is encapsulated by the margin of error which is added or subtracted from the sample value obtained in a poll. Mathematically, this can be defined as:

Population value = sample value ± margin of error (± means add or subtract)

In the case of an EU referendum poll, the sample value would be the proportion of the sample that vote Remain or Leave while the margin of error provides the space to capture the proportion of the population that would vote Remain or Leave. This margin of error is set to 3% in most polls which is not reported by mass media. Hence, readers might erroneously assume that the polling results represent the true proportions of the population voting in a particular way. Although the margin of error is an essential tool to account for variations of the population value from a poll, it is insufficient as readers cannot comprehend the different outcomes that might be generated. This can be resolved by using confidence intervals.

A better tool for reporting variation in results: confidence intervals!

We can subtract or add the margin of error from the sample value to produce the lower and upper bounds of a population value respectively. Combining these bounds produces confidence intervals, the range of values that we are almost certain captures the true population value. Although they can be produced at varying levels of confidence, they are usually set at 95% confidence to almost guarantee (at 95% confidence) that what we conclude from a poll can be generalised to the whole population. This makes it very useful for thinking about the different outcomes of the EU referendum that might be generated from conducting a poll.

Calculating the confidence interval of a polling result

1. Find the sample value of a poll. In our case, we want to calculate the proportion of the sample that vote Leave. This can be calculated as:

2. Calculate the margin of error (MOE) to measure variation in the poll results. In our case, to calculate the MOE for our 95% confidence interval, we use the formula:

The MOE is influenced by the sample size which describes the total number of people that have voted (Total(voters)) in the poll.

3. Subtract or add the MOE from the sample value to get the lower and upper bounds of the confidence interval respectively.

Lower bound = sample value – margin of error

Upper bound = sample value + margin of error

4. Combine the lower and upper bound values to generate the 95% confidence interval. This describes the range of values that we are 95% sure captures the true population value (in our case, the proportion of the population that vote Leave).  

Confidence interval = (lower bound, upper bound)

The basics of hypothesis testing

Although confidence intervals can describe the different outcomes of the EU referendum, it does not give a clear-cut answer of whether we should care about a poll. This is where hypothesis testing comes in.

In hypothesis testing, we assess assumptions of a population value against data of a random sample. It is analogous to a criminal trial, where a criminal is presumed innocent until enough evidence is collected to prove guilt. In the same sense, we assume that the null (H₀) hypothesis is true until proven otherwise. The null hypothesis proposes that there is no deviation from a set population value in response to some event. This arises when we produce a polling result that would have most likely appeared by chance. Conversely, if we have a polling result that is so rare and unusual that it would not have arisen by chance, then we have enough evidence to reject the null hypothesis and accept the alternative (H_a) hypothesis. The alternative hypothesis describes a deviation of the population value from some set value.

In our case, we want to assess whether a poll can decisively conclude that most of the population would vote Remain or Leave. We write our two hypotheses as follows (prop_0,Leave is the proportion of the population that would vote Leave from the null hypothesis):

H₀: there is an even split of Remain and Leave voters in the poll.  prop_0,Leave = 0.5

H_a: the poll decisively favours Remain or Leave. prop_0,Leave ≠ 0.5

Should we care about a polling result? Let’s use a hypothesis test to find out!

There are many statistical tests that can be used depending on the kind of data that we are analysing. As we are analysing the proportion of people that vote Remain or Leave in a poll, we convert the proportion to a standardised z-value that can be used to calculate probabilities on a normal z-distribution (better known as a “bell curve”).

The z-value can be calculated by the formula:

If we set prop_0,Leave = 0.5 (meaning an even split of Remain and Leave voters in the population), we can simplify the z-value to:

This z-distribution (Z) can be used to calculate the probability (the p-value) that we generate a random result that is just as or more extreme that the polling result given some set value from a null hypothesis. This is represented mathematically as:

And can be calculated using normal tables, a calculator or a computer. We compare the p-value to an alpha value which is the threshold that the p-value has to go below to reject the null hypothesis. Although we can set different alpha-values between 0 and 1, it is usually set to 0.05 (which describes a 5% chance that we get a random result that is just as or more extreme than the polling result given some null value).

• If the p-value is more than the alpha value (i.e., p > 0.05), then we have failed to reject the null hypothesis. We conclude that the poll cannot decide whether most of the population would vote Remain or Leave in the EU referendum.
• If the p-value falls below the alpha value (i.e., p < 0.05), then we reject the null hypothesis and accept the alternative hypothesis. We conclude that the poll can decisively favour Remain or Leave in the EU referendum among the population.

Hypothesis testing is useful for deciding whether the public and stakeholders should care about a polling result, facilitating informed decisions on how campaigning needs to be done.

Applying hypothesis testing and confidence intervals to a real-life EU referendum poll

Let’s look at an online poll run from 27^th to 29^th May 2016 by the polling company ICM. Out of 1753 people, 848 (48.37%) voted Remain and 905 (51.63%) voted Leave. Should we care about the ICM poll?

First, let’s use hypothesis testing to decide whether the ICM poll is decisive. We declare two hypotheses:

H₀: There is an even split of Remain and Leave voters in the population. prop_0,Leave = 0.5

H_a: There are more Remain voters in the population than Leave voters. prop_0,Leave ≠ 0.5

Since we have prop_Leave = 0.5163 (converted from percentage to decimal), we calculate the z-value as follows:

And calculate its p-value:

The p-value of 0.1738 exceeds the alpha-value of 0.05, so we failed to reject the null hypothesis. The ICM poll cannot decisively favour Remain or Leave, implying an even split of two sides among voters in the population.

How can we visualise the indecisiveness of this poll? We can use confidence intervals to do this.

First, calculate the margin of error (MOE). The MOE will be the same regardless of whether the proportions of Remain or Leave voters are used.

This is within the 3% MOE mentioned in most polls.

We use the MOE to calculate the confidence intervals of Leave and Remain voters.

Leave confidence interval = 51.63 ± 2.34% = (49.29%, 53.97%). This confidence interval states that we are 95% sure that the true proportion of the population that would vote Leave is between 49.29% and 53.97%.

Remain confidence interval = 48.37 ± 2.34% = (46.03%, 50.71%). This confidence interval states that we are 95% sure that the true proportion of the population that would vote Remain is between 46.03% and 50.71%.

These confidence intervals appreciate that the proportion of the population voting Remain or Leave might differ from the polling results. The real power of confidence intervals; though, comes when we visualise them in a number line.

The number lines above show the ICM poll results (indicated by the middle point of the line) along with the Leave and Remain confidence intervals. Two things can be observed from the number line:

1. A 50:50 split between Leave and Remain voters is possible in an EU referendum (indicated by a dashed line) because the confidence intervals of both the Leave and Remain sides contain the 50% proportion. This result would not provide a clear indication of which side would win, something the mass media does not appreciate when hyping up a particular result.
2. A referendum involving the population might produce a different result from a poll. Although the poll had a higher proportion of Leave than Remain voters in the sample, it is possible that in a referendum over the population, there might be more Remain than Leave voters. Hence, the poll cannot conclusively favour one side over the other.

These two points open the possibility that the poll might not capture the views of the population. This is something the reader overlooks not only because the mass media excludes the margin of error but because they do not realise that the polling results may not reflect the views of the whole population. If the confidence intervals of two groups in a sample overlap each other, it is possible that the referendum results of a population might be very different from the polling results of a sample.

Conclusion

How polling results are reported by the mass media today covers up the dangers of extending results from a sample to infer conclusions about a population. Even citing the margin of error does not paint a true picture of the range of possibilities that might arise from a poll. In contrast, hypothesis testing and confidence intervals can produce a lot of insights of how we interpret polls. While hypothesis testing can tell us whether we should care about a polling result, confidence intervals can reveal the variability produced when polling results are extended to the overall population.

Ideally, the mass media would adopt hypothesis testing and confidence intervals as tools to correctly interpret polls and to responsibly extend results to the population. Given the mass media’s interest in hyping up polling results regardless of whether they are warranted or not, this is most likely not possible. Hence, independent companies should be set up to analyse polling results and to provide a truthful interpretation of the polls to the public so that they can decide whether they should act on a poll or not. Keeping the polling industry accountable to these statistical measures will ensure the viability of polls in painting a truthful picture of how the population thinks on various issues of the country.