Superforecasting

As modellers, we can no longer sit back and take no responsibility for the forecasts made with our models

A number of high-profile infrastructure projects in the UK (CrossRail 2, airport expansion, HS2) will have their demand forecasts scrutinised at a greater level of detail and more publicly than we are generally used to

A number of high-profile infrastructure projects in the UK (CrossRail 2, airport expansion, HS2) will have their demand forecasts scrutinised at a greater level of detail and more publicly than we are generally used to

If you had asked me five years ago what the main role of a transport modeller is, I would have said: 'to develop and use good transport models, but not to determine the required inputs about the future'. In my view that would have been the job of another expert, an economist perhaps, or a demographer. I would have said that good transport models would obviously show a good fit to observed data, and that we as modellers should be able to show that, when forecasting, the output elasticities were reasonable (the kinds of tests we have for showing such reasonableness are quite limited at the moment, but that is a discussion for another day).

I would have said quite firmly that we as modellers cannot take responsibility for the forecasting of input parameters (such as population, economic or technological development). Our job was to create and apply the tool diligently – we had and should have, as the saying goes, no skin in the game, ensuring objectivity.  If forecasts turned out wrong, due to erroneous future input assumptions that had been provided by others, it wasn’t our fault.

Nowadays I feel quite different. There are plenty and recent examples of international model forecasts that were way off outturn flows. A number of high-profile infrastructure projects here in the UK (CrossRail 2, airport expansion, HS2) will have their demand forecasts scrutinised at a greater level of detail and more publicly than we are generally used to. The debate about the impact of technological transformation and behavioural change on travel demand has only just started and illustrates an uncertainty about the future inputs to our models that is much greater than we have traditionally experienced. As modellers we can no longer sit back and take no responsibility for the forecasts made with our models.

First, poor forecasting using our carefully crafted models brings our profession into unnecessary disrepute.  The problems with some of the toll roads forecasts gone wrong in Australia have tainted the modelling profession as a whole; and most calls made to punish modellers for poor forecasts, which so outraged many in our profession, were based on forecasts having been made by those that did not necessarily develop the original tools, with the modellers no longer controlling the input parameters.

Second, most of our clients don’t want to buy just a model, they want to buy a forecast, or forecasts. An analogy – you don’t want to hire a plumber in general, but you want your leaking radiator fixed (admittedly by a qualified professional with a proper toolkit).  Too often, we as modellers run out of time or budget to derive the real decision support value out of the efforts and costs expended on model development, a single forecast is made and the tool gathers dust. Job done. I am not saying that our clients want a particular outcome from our forecasts – what I mean is that it is the application of the model that creates the value, not the development. 

Third, who would be in a better position to use the models that we built, than us? We know their strengths and weaknesses, the underlying assumptions, any data issues, the idiosyncrasies in the underpinning theory.  By taking a more active interest in inputs about the future, we can at least ensure that these inputs are as good as possible within the model’s theoretical and practical constraints.

Finally, I can imagine a future in which transport modelling will no longer involve base year estimation – this would be done automatically through the use of big data – and our sole raison d’etre would be the development and intelligent application of the forecasting elements in our models.  This would involve the development of future input scenarios, based on our understanding of the current transport system and the society it supports and possible future social, economic and technological developments. Ultimately this may be about survival of our profession;  the elements that we concentrate on now might be done – and done better – through artificial rather than human intelligence, by data-driven methods and computers. The things we shy away from now may well be where we can actually continue to make a difference in terms of evidence based decision-making.

My thinking about our roles as transport modellers in shaping forecasts was influenced by a recent Freakonomics Radio Podcast: 'How to be less terrible at predicting the future'. I was obviously intrigued and can recommend this podcast to you – take an hour out, grab a coffee and sit back…  Although the predictions discussed in the podcast are much more related to politics, economics or sports than to transport policy or infrastructure demand forecasting, I did pick up a few pointers that are relevant to our own profession.

People are bad at forecasting because there is no danger in getting it wrong. That is sometimes true for us modellers as well. Even if our forecasts look wrong, we are powerless, as the future growth assumptions are prescribed by third parties including Government (but our models are surely correct...).  But linking back to the second point above – we were not asked to produce (just) a robust model, we were also expected to provide a decent forecast.

A bad forecaster is systematically over-confident and often dogmatic in his/her view of the world, unwilling to change his/her mind in response to new evidence. Good forecasters (the podcast goes as far as identifying the traits of superforecasters) are humble about their judgments, are curious and are willing to challenge conventional wisdom. Both Transport for London and the Department for Transport are starting to contest some of the basic behavioural assumptions that have traditionally underpinned transport models, even though this has not yet filtered through into models themselves.  

The average forecast derived by a group of forecasters tends to be more accurate than most of the individual forecasts from which the average was derived; we are more likely to provide better forecasts if we are willing to learn from other experts. This, of course, has been Modelling World’s aim from the start!

As a profession, we have not been in the habit of keeping score, we haven’t evaluated enough the out-turn accuracy of our predictions, we tend to find reasons why the model was right but the inputs were wrong, and we miss out on valuable learning opportunities. But will we ever become superforecasters? 

Tom van Vuren
Chairman of Modelling World, 
Divisional Director at Mott MacDonald and a Visiting Professor at the University of Leeds

Tom Van Vuren is speaking at Modelling World 2016