Conometricks and the new NAFF ‘case’ for HS2

Mark Twain contended that there are three kinds of lies: 'lies, damned lies, and statistics'. These matters have become even more vexed since his sardonic remark in 1906. Whilst the (potential) sophistication of quantitative economic analysis has increased considerably, so also have the possibilities of dissimulation via the deployment of sophistical econometrics and other means of quantitative economic bamboozlement. I label these latter groups of wily economic arts collectively as Conometricks. Their common feature is that they all involve the insidious warping and/or specious misuse of (often complex) statistical/quantitative analysis so as to bolster a weak (or dire) economic case, argument, or projection.

A major matter in this regard in the UK concerns the scale and variety of conometricks that have been deployed since 2009 to cobble together some form of ‘economic’ justification for the proposed High Speed 2 (HS2) mega-project. It lacks any commercial rationale.

How, then, to justify an ‘economic’ case for this political vanity project? In 2009 the government set up an offshoot of the Department for Transport (DfT), with staff seconded from the DfT (and from Network Rail, another quango) and called this progeny HS2 Ltd. This organisation was tasked with objectively appraising the ‘business/economic’ case for HS2. Not unsurprisingly, given that HS2 was its only project (and thus its raison d’etre), the new HS2 Ltd quickly divested itself of the tiresome hobble of civil service objectivity. It became instead the dispenser/organiser of a vast raft of government contracts, doled out to a variety of consultancy firms to, effectively, concoct a case for HS2.

On this basis, during 2010/11 both HS2 Ltd and the DfT (which are effectively the same entity) published studies/reports claiming that the estimated ‘Benefit/Cost Ratio’ [BCR] of the HS2 Project was generally of the order of around 2.4 to 2.7 (depending on whether ‘Wider Economic Benefits’ [WEI] were included; or not). All of this sounded quite reasonable on HM Treasury Green Book criteria; which demand a (clearly-identified) BCR above 1.6 (at absolute minimum).

Subsequent evaluations by independent economists revealed, however, that this so-called business case had been built upon a motley variety of extremely questionable assumptions, including:

  • Predication of the analysis upon an (assumed) high growth rate for the UK economy for many decades to come (up to 75 years!);
  • A numerically high - in the long run clearly fallacious – assumption about how the (supposedly) fast-rising incomes in the UK would convert into extra intercity rail travel;
  • Avoidance of the question of what the price of the (proposed) HS2 rail travel in the UK might be (e.g. premium, as with HS1); and, how consumers would react to that;
  • Avoidance of the evaluation of alternatives - including both road and rail alternatives to HS2;
  • Avoidance of the question about how new developments in ICT might affect the demand for (high-speed) rail travel in the UK over coming decades;
  • Avoidance of the matter of the opportunity costs of this (vast) public investment; e.g. as regards other, alternative, forms of communication and connectivity - for example, broadband (let alone roads, or alternative rail).
  • Avoidance of the matter, in the cost elements of the BCR calculations, of planning blight; and disruption costs during its (lengthy) gestation.
  • The assumption that the time spent by business travellers on intercity trains was of zero productive value, which greatly exaggerated the calculated benefits of higher-speed rail.

In short, the initial HS2 Ltd/DfT ‘economic case’ for HS2 involved a (big) bag of conometricks! The indications are, however, from the independent evaluations noted earlier, that the true/sober BCRs for HS2 could actually be below unity.

What has been the government's response? The DfT (grudgingly) accepted that the HS2 BCRs may be ‘lower’ than it previously proclaimed. However, it is now also trumpets a new line of argumentation about HS2.

We're now told to ignore the previously highlighted arguments (about speed/time savings), and concentrate on ‘capacity and connectivity’. According to the Secretary of State for Transport (11/9/2013), Patrick McLoughlin, transport capacity between the North and South of England will be completely ‘overwhelmed’ within 30 years unless this (very specific; no alternative provision is considered, or evaluated) investment in the HS2 project is made.

The tenor (and content) of Mr McLoughlin’s apocalyptic diagnosis is highly reminiscent of a forecast made in 1894, by a (London) Times writer - working on the basis of ‘precise’ calculations - that London would be literally overwhelmed by horse manure by 1944 (at the very latest).3 Obviously, this forecast did not come about - as alternative ‘connectivities’ to horse- drawn vehicles (e.g. the car, cycles, the bus, rail, tubes, telephones etc.) were developed.

On the very same day as the Secretary of State made his ‘overwhelming-of-England-in-30- years’ prophecy, HS2 Ltd published a new analysis of related matters conducted for them by KPMG (a consultancy firm). This study claimed that HS2 could - by releasing capacity on the old network; and increasing intercity ‘connectivity’ - boost UK GDP by around £15bn p.a. (but only after 2037…something ignored in many media reports).

Whilst the study waffles on about ‘forecasting the future’, it also makes plain, in an upfront disclosure (p.ii), that its analysis does '...not constitute a forecast' (emphasis added). We have here, then, a classic example of a Not-A-Forecast Forecast; or a NAFF, as I call them for short. [NAFFs are actually quite common: they are otherwise known as ‘deniable’ forecasts: the forecasters involved deny responsibility for their own forecast).

Like its predecessors, the KPMG study also neglects alternative means of achieving ‘connectivity’ (e.g. ICT; roads; other rail); and fails even to compare and evaluate its preferred ‘solution’ against alternative possible rail improvements. And buried in its Technical Appendix (at 6.3.38; p.83; if you can stick with it that far!) the study accepts that a key element of its approach 'does not have a firm statistical foundation’.

The real conclusion is that the KPMG 'technique' employed ‘captures Sweet Fanny Adams’. Indeed Professor Henry Overman of the LSE (and a sometime panel adviser to HS2 Ltd, until June 2012) has suggested that key aspects of the KPMG methodology are ‘technically wrong’ and that, in his own analyses, properly accounting for the effect of other variables had reduced the impact of rail connectivity by six to eight times.

Despite all of this we are told that ‘Ministers are due to publish an updated case for the (HS2) line, drawing on a variety of new work; including the KPMG report’ (emphasis added). It therefore looks like we’re in for another round of taxpayer-financed, pro-HS2 shenanigans coming along soon. A government spokesman has said that ‘the case for HS2 will be improved further, when we publish shortly the updated strategic case…’

The (all-party) Treasury Select Committee (TSC) has, however, (rightly) expressed serious concerns about the quality of the BCR analysis conducted to date to support HS2; and called for the Treasury itself to evaluate the claims put forward by HS2 Ltd and the DFT. I would go even further: I think the UK Statistics Authority should take a good, hard, look at the ‘conometricks’ that have been deployed to support HS2 - at very considerable public expense.

When I was writing 'They Meant Well', a Hobart Paper about government project disasters, I must admit I was surprised to come across a 2002 study by Flyvbjerg et al which concluded that cost overruns are best explained not by error but by 'strategic representation' -- in other words lying -- in order to get projects started. I'm afraid that's exactly what is happening with HS2. John may prefer to call it 'Conometricks'. At all events it seems clear that politicians and others are deliberately not telling the (whole) truth in order to try to build plausibility into their 'passion' for HS2. It is a disgracefully irresponsible proposed use of taxpayers money. The only thing I can find in their favour is that at least they do seem to have included a decent chunk of contingency allowance, as they certainly should (though, according to Richard Wellings, not nearly enough).
Your methodology and dodgy accounting was savaged by members of all parties in the House of Lords this week. Do you think that those in glass houses should be careful of throwing stones?

@Chris - I had the misfortune to listen to some of yesterday's debate. It was clear to me that the Lords concerned either hadn't read the paper at all or at least hadn't read it properly. My impression was that they were parroting briefings from third parties in their misleading and inaccurate comments.

Since its publication in August, several developments have vindicated the paper, with announcements of substantial additional tunnelling to buy-off local opposition, large-scale taxpayer subsidised regeneration schemes, and new transport infrastructure to make cities 'HS2 ready'. These extra costs are exactly as predicted in the analysis. The recent 'logrolling' behaviour of officials in London is also noteworthy.

re. Chris's comment above: Chris, you are not challenging my analytical critique at all; but I presume , referring to Dr. Wellings study... perhaps you could address yourself to what I have said?...if you intend to make your comments pertinent to the posting?
Excellent piece - timely, short and compelling. Four thoughts: (1) Britain's railway system was built in the 19th century entirely without subsidy. There may indeed have been significant reverse subsides in the form of bribes paid by the railway companies to MPs. (2) I am extremely sceptical that there are any significant benefits not captured by the market. if there are then for sure there will be similar benefits in the foregone alternatives. (3) The whole thing stinks of the 'appraisal optimism' that was the ruination of the old nationalised industries. (4) If it is such a good idea why don't the civil servants and politicians invest, say, 10% of their pensions in it. Robert Miller

Post new comment

The content of this field is kept private and will not be shown publicly.
Type the characters you see in this picture. (verify using audio)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.