Fantastic modellers

Are modellers fantastic – or living in a land of fantasy? Tom van Vuren defends his peers....

Anyone who subscribes to David Levinson’s Transportist newsletter cannot have missed his recent headline of “Fantasy Modeling”. Inevitably a debate has ensued on social media as to whether his observations and accusations of models and modellers (“these models can lead to the misallocation of public funds by overstating (or understating) the benefits of transport projects, and ("they are often not conducted in good faith)” are well-founded or not.

Every now and then, and far more often than I would like to see, an article surfaces that attacks transport modelling and forecasting, and harshly. In 2022, Create Streets published their Briefing Paper “Computer Says Road”, with the accusatory subheading “why outdated transport models ruin new developments and how to fix them”.

---

Join the debate: more modelling discussions will be held at Modelling World on 2-3 November: Book your place now

---

Many of our planning colleagues who’d rather see us dispense of building evidence for decision-making using models and demand forecasting completely, for example academic Bent Flyvbjerg, Emeritus Professor, Said Business School, University of Oxford, who doesn’t mince his words. 

In a 2012 blog by Elizabeth Harrin, Flyvbjerg is quoted saying that “the majority of forecasters are fools or liars” (I haven’t been able to find this reference myself, though).

Further quotes attributed to Flyvbjerg in the blog are that “some estimates are deliberately misleading”, whereas the press release accompanying a 2013 paper in the International Journal of Project Management apparently had him say: “Many forecasts are garbage and can be shown to be worse than garbage”. I have invited Flyvbjerg in the past to come and speak at Modelling World – an invitation he has not yet accepted.

In the 2013 paper, Flyvbjerg sets out his position: “the front-end estimates of costs and benefits – used in the business cases, cost–benefit analyses, and social and environmental impact assessments that typically support decisions on projects – are commonly significantly different from actual ex post costs and benefits, and are therefore poor predictors of the actual value and viability of projects”. 

And although the former is undeniably true, the latter is to my mind unproven: although project outcomes are generally different to our forecasts, the use of carefully constructed and estimated models to explore the need for and value of interventions is surely better than a guess, even an educated one. 

Despite his provocative position, his suggested improvements make sense, and should also in my mind be part of the standard forecasting processes, particularly for large-scale projects. Flyvbjerg promotes what he calls ‘the outside view,’ to mitigate optimism bias and other biases, because it bypasses these by cutting directly to the empirical outcomes (rather than the processes). 

Strong peer review can deliver that. A further point he makes is the need for due diligence in the form of benchmarking (in my words, checking the modelled results against common sense based on similar projects elsewhere).

Obviously, applying good practice, such as set out in the DfT Transport Analysis Guidance (TAG) or in the Australian ATAP guidelines, and adhering to ethical standards by professional organisations such as the Transport Planning Society, CIHT, and AITPM, the Australian Institute of Traffic Planning and Management, should reduce (or I’d like to say eliminate) any shadow of ethical doubt. In support of that view, I particularly like the statement in TAG Unit M2.1 (paragraph 6.6.2): 

Sensitivity testing should be undertaken of the model’s behaviour against variation in those parameters that are judged to: 

• have a substantial effect on the model’s prediction of changes when forecasting, and 

• be uncertain in their calibration.

In my LinkedIn response to Levinson’s challenge, I repeat a point that I have made many times before: in terms of the impacts of model results on good or bad transport infrastructure decisions, we need to separate the roles and responsibilities of a) the models and their developers, b) the modellers who use the models for forecasting, c) the model sponsors commissioning the model forecasts, d) the economist end users of the model outputs to calculate (dis)benefits of the investment and e) the decision-makers who use model results as part of their responsibilities.

If we want to reduce singling out of models and modellers as the root of all transport infrastructure evil, we have a role to play in bringing all these people together, and onto the same page.

There also a lot less reason for sceptics to criticise our demand forecasts if:

a) sensitivity tests and reasonableness tests have been carried out and reported as per TAG guidelines and 

b) assumptions about the future world, in terms of all drivers of demand such as the economy, network and land use assumptions, and including possible future changes in travel behaviour, have been agreed with promotors and the wider stakeholder community, carefully reported, and the associated uncertainty on project outcomes quantified. 

Then it’s up to the decision-makers to choose the weight they put on our model results, compared to all other evidence.

Levinson summarised the comments to his initial provocation in a follow up post in his Transportist newsletter called Fantasy Forecasting Follow Up (and I’m uncertain if he chose the word forecasting rather than modelling for its alliteration or because what he really is concerned about is the forecasts rather than the models underpinning them).

Read them - we should be willing to listen, and not ignore the messages and suggestions for improvement. 

And if you want to become an even more fantastic modeller than you undoubtedly already are, consider joining the Society for Decision making Under Deep Uncertainty – it’s free!– and learn from their approaches to handling the challenges of long-term forecasts.

---

Tom van Vuren is Chairman of Modelling World, a Visiting Professor at the University of Leeds, Policy Director at the Transport Planning Society and a Strategic Business Partner at Amey