Many have expressed the opinion that the very simple correlations shown in The Spirit Level alone can never show meaningful causality – this is correct. However, people can always claim the academic literature shows the same thing. Unfortunately this gulf will always exist, and the public (and even occasionally other academics) can be tricked into believing a relationship, because there is a so called academic ‘consensus’.
In this article I will present 4 simple questions everyone should ask before they believe the results of some revolutionary new study that claims x is causing y, including any of the claims made in The Spirit Level. In this context I will also explain the results of my recent paper in Economic Affairs, which is probably the first analysis to use UK data to test one of the claims of The Spirit Level. In particular, we investigated whether regional and district level inequality (Gini coefficient) determines individual likelihood of obesity.
% obese men vs. inequality measure
Firstly, is the study adequately designed to answer the question? In other words, does a graph like the one above, even if it accounts for all the problems I discuss below, help us discover whether living in a more unequal area increases your likelihood of being obese?
We know what roughly determines weight at the individual biological level (genetics, age, gender, energy consumption and energy expenditure) and the suggested relationship goes from higher inequality to feelings of low status, which are supposed to induce behaviours such as ‘comfort eating’, which then increases weight.
If this is the theorised causal sequence involved, then it makes far more sense to sample individuals and see how individual weight (i.e. BMI) differs according to the inequality in their geographical area. In fact as I explain in the paper, even the other academic studies that look at this issue using a cross-country approach suggest the next stage of research should be of this sort. This is what economists call multi-level models and this is the approach we take.
Secondly, is there a third (fourth? fifth? tenth?) variable driving the relationship? More precisely, are there ‘confounding’ factors determining x and y so that there only appears to be a causal relationship? Economists use a statistical tool called multiple regression analysis to help ‘control’ for these other factors. The most intuitive way to ‘control’ for stuff would be to go out and collect a sample of people that were identical in terms of all possible confounders and then see what relationship there is between x and y – this is very impractical so instead we collect a random sample and imitate this process with regression analysis.
The Wilkinson and Pickett paper from which the figure is taken, controls for gross national income per capita – but why not control for average aggregate versions of all our controls, among a variety other things (e.g. poverty levels)? Some of the nine multi-level studies we found lacked key controls: for example two studies did not control for individuals’ education and three seemed to be using imperfect proxies for individual or family income. Yet these are well known determinants of BMI and may be confounders – e.g. maybe being poor implies there are more poor people around you which translates into a higher regional/district Gini?
Informed by the decades of research into the individual determinants of obesity (and with some intuition), we chose a wide range of controls for our study which should help overcome some of these issues. There could always be confounding factors we have missed, but improving the number of controls helps us get as close as possible to a sense of causality.
Thirdly, is there reverse causation in the relationship? The simple case of this is that y is actually determining x, the converse of which should not show up if we have a good set of controls. Otherwise this is a particularly difficult issue statistically: even if we were to have perfectly controlled for everything we will not be able to isolate the effect of x on y, if y is also simultaneously determining x. Economists call this an equilibrium process and a classic example is the level of crime and police in an area. Fortunately, because it is very unlikely that an individual’s weight is determining inequality in their area, this should not be a problem for our study. But is it possible in aggregate studies, of the kind shown in the figure? It does not seem implausible.
Fourth: is there a statistically significant relationship? What if I repeated my study identically but instead of using a sample of around 10,000 we only sampled 10 people, and found that inequality has a very large effect on BMI (i.e. a high ‘economic’ significance). Would you be convinced the relationship holds for the population? Not really - this is why economists have to consider the somewhat subjective notion of statistical significance, as well as economic significance. The likelihood of statistical significance rises as the effect size increases or the sample size increases. Remember when observing any of the graphs in The Spirit Level, there is usually no indication of statistical significance.
All effects we found were statistically insignificant at commonly used benchmarks, except for some evidence of a district-level effect, but this was economically insignificant compared to other determinants of weight. Results across other studies were also mixed.
In conclusion, the reader should ask these 4 questions when confronted with any statistical claims - including any of the graphs from The Spirit Level. Further still, we can say that there is currently not enough evidence to suggest a meaningful relationship between inequality and obesity for the UK – or for that matter any country.
Darshan Zala is the author of ‘Challenging The Spirit Level: Is There Really a Relationship between Inequality and Obesity?, published in the June 2013 issue of Economic Affairs.
Post new comment