Shaking the foundations of modelling: Luis Willumsen

Luis (Pilo) Willumsen discusses the opportunities and challenges he foresees in the next decade and beyond

The 2007/2008 banking crises was very significant and influenced my view of modelling. I was working with some of the best banks in the world on a toll scheme and requested economic forecasts for the country, which after some reluctance they provided. These were optimistic: the world was going to continue its high speed growth; however, a few months later Lehman Bros collapsed. The banks were blind to the looming crisis, and that was a big shock to me. We know that our forecasts as modellers are not always accurate, but the fact that the best banks in the world could not foresee an imminent crisis, mostly of their own making, was very significant. It shook my assumptions about forecasting.

Luis will be at Modelling World 2015 on June 4: www.ModellingWorld2015.co.uk

I became preoccupied with two things: the nature of forecasts and the theory underpinning our models. 

I believe that outside the academic world the objective of modelling is in essence forecasting. If so, we must ask the question: how much can we actually forecast the future? We’re always in the hands of external events that can change contexts in a significant way. What is the point, then, of providing a single point forecast; the only thing certain about this single forecast is that is bound to be wrong. I believe that in forecasting we must embrace the unavoidable uncertainty of our projections. This should be a central focus of any effort to improve our models and forecasting abilities. 

The irrational behaviour that created the global financial crisis demonstrated that as human being we are only partially rational. Behavioural Economists and Social Scientists knew that for years, but our best models still assume rationality in general and, in most cases, perfect information. These are no longer tenable. The real human being is not the one assumed by current models: we value losses more than the equivalent gains (asymmetric elasticities); we are more aware of relative changes than absolute values (Logit with linear in the parameters utility functions is wrong); it takes time to make some choices (our models ignore lags and hysteresis) and we can never consider more than seven (+/- 2) alternatives available to us. All of these, and more, are realities ignored in our models and this is a gap we must try to close in the next few years. If we cannot close it we must enhance model outputs with interpretation and judgment, and this should be accepted as valid.

We need to improve our models to be able to cope with uncertainty and risk. Our emphasis on equilibrium, guarantees that we’re making fair comparisons between alternatives. However, these fair comparisons are made on key performance indicators (KPI) that are poorly estimated in a world with many uncertainties. We need to do better than that. For an uncertain future we need to understand what future scenarios may actually materialise and test our forecasts in them. Some of these scenarios are going to be radically different from what we have now. Self-driving cars, for example, could become very popular very quickly rather than slowly as some forecast, and we don’t really know. Which is the better choice for investment when faced with different scenarios,  different futures?

The impact of the banking crisis was not just on the economy, but also on the stability of model parameters. Our models had to be adapted to handle this: the values of time changed because of a change in sentiment, in mood, rather than because the economy was collapsing. There was a psychological effect on the value of time. Income, value of time and willingness to pay are very closely related. For some people, income didn’t change but their willingness to pay did because of a fear for the future; that they could no longer continue spending money in the same fashion.  People became savers rather than spenders. This was notable even in Australia, which didn’t suffer much from the recession. These things have shaken some of the foundations of what we do with modelling, and significantly changed my mind about our assumptions relating to rational behaviour. 

Several creative thinkers penned their thoughts around 2008 in books such as Predictably Irrational by Dan Ariely and Thinking Fast and Slow by Daniel Kahneman. These should be compulsory material in any Transport Modelling course; they emphasise the fact that we are not always rational, but our models assume that we know exactly what we are doing. But such assumptions are no longer tenable. For the time being, the models we have are our best, albeit imperfect, tools. We must complement these with better interpretation and judgment on their outputs. This puts the modeller and forecaster centre stage as we can no longer hide behind the quality of our model calibration/validation. This is certainly more risky for us as modellers, but in my view it is also more honest and transparent. If we cannot forecast accurately, as I think is impossible for projections beyond the next five years, we should at least provide good judgement and decision-making advice in respect of possible futures.

Clients and decision-makers, however, may not warm to such an approach. Many want a single answer, not a range of answers. Many are stuck in the same old mind set, and it's going to take some time for this thinking to evolve and mature. On the other hand, when dealing with traffic and revenue forecasting for the transport concessions, many clients, especially outside of the UK, are much more open to explore the impact of alternative futures and unavoidable uncertainties. Where people are taking risks with real money, they want to understand better what the uncertainties are, whereas the public sector is, at the moment, perhaps less interested.  

Future challenge one: Big data

?For at least 40 years we have been modelling with a lot of theory (not always that good) and little, and often poor, data. The future is different. There are now a plethora of sensors generating valuable data for our potential use in modelling and forecasting. This creates significant opportunities and major challenges. One issue is that this new data may help to recognise previously invisible correlations. Disclosure: I am a director of a company set up to process these new sources of data. 

I have been involved with data collection using anonymised mobile phone data. It's clear that we can begin to understand two things which beforehand we didn’t have any insight into. One of them is to know what actually happened yesterday.  Previously, without planned surveys or sensors on the road, we had no way of knowing what happened on a specific date. 

It's interesting to find out what happened on a particular day without having to plan the data collection for that day. This is particularly useful in terms of learning from unplanned experiments, such as flooding or sudden strikes, about how mobility changed as a result, and about travel flexibility in the short term. 

A second unique opportunity is to look at trips in a different way in terms of their repeatability.  As we depend on data collected for one day we tend to assume that the observed and expanded trips are actually repeated on this “representative” day. However, we now know that many trips are irregular. 

We have learned that there are fewer repetitive regular trips than we once believed, and our models assume. Until now we had no way of collecting such data and making models or decisions around the difference between regular repetitive trips and irregular occasional trips, and this is something that may be more important that trip purposes other than to work and education. 

Perhaps there is nothing in it, but I suspect that there is. With urban toll roads, for example, only some 25% of the trips are repetitive such as commuting; all the others are occasional, opportunistic, trips. The guy is willing to pay because on that particular day in that particular location he was in a hurry and 10 minutes saved on a toll road are worth more than the toll. 

So again, the fixed, uniform value of time may no longer be useful. There are varying levels of willingness to pay depending on the occasion, the trip, the time and the conditions. We will have much richer datasets in the future.  Hopefully some of these will be related to uncertainty, the reliability of travel time, but as yet we simply don’t know. This is a new research area, that was not possible yesterday because of our limited data collection opportunities. For me, this is very interesting and will lead to some big changes in modelling and forecasting in the next 10 years. 

Future challenge two: Autonomous vehicles

Self-driving vehicles will also change the way we think about transport and how we model it. I think that we will begin to see these in regular use somewhere between 2020 and 2025. Audi, Mercedes Benz and the Googles of this world believe it's going to happen earlier, around 2018. But that’s only one question; how quickly will self-driving vehicles catch on will be determined by people. The second question, which is equally important, is how will the change happen? Will we have 95% private car ownership, or will rental models prevail? There are some good reasons why it's very likely that many people would prefer to rent such vehicles, by the hour or minute. First, as an insurance against an even better technology turning up in a year, and second, as it will be cheaper in terms of whole life costs to rent than to own; sharing depreciation among multiple users and not having to worry about maintenance, MOTs and obsolescence. A strong reason in big cities will be to eliminate the need to find a parking space; use the car and abandon it at destination to serve the next customer. Finally, users could rent the car they need on each occasion: city car, pick up truck, 4x4, and not buy an SUV that they use as such five weekends in a year.

This changes everything, because one car will serve 5 or 10 people in a day. Levels of congestion will change, we don’t know exactly how, as will levels of perceived cost to the potential car user. There are some very interesting questions around this. If you are being driven automatically by your self-driving car, your time will no longer be 'wasted'. You could be talking on the phone or playing games, working, reading or thinking, and everyone will benefit from that.

It will challenge public transport. Rail public transport is difficult to beat, as it's generally fast, reliable and comfortable; I don’t think it will be threatened by self-driving cars. But low frequency buses will be at risk due to increased the convenience that self driving cars will offer. Taxis will be threatened and will eventually fade away, unless it’s an Uber-style model; these companies will be very well placed to organise self-driving cars and work with them.

A need for honesty?

In the last few years I started working more as a peer reviewer. Looking at somebody else’s model is enlightening, as in a way you are actually searching for problems. And there are problems. I am now convinced that every model under the sun has errors. The more complex it is and the more errors it has, from coding errors to scripting bugs, even simply spreadsheet mistakes.  These errors seem to be no insurmountable obstacle to good calibration and validation. However, these errors increase the likelihood that outputs will be distorted.  We need to step back, think on how much we understand about human behaviour, and how much of that individual behaviour is actually forecastable, to decide on the complexity required. 

I don’t think there is a lot of advantage at this stage in making models more complex; indeed, there will be advantages from making them simpler. It's better to be able to run more alternative scenarios rather than adding layers and layers of complexity. Each new layer may be justified, because it contains a little bit more about human and travel behaviour. But the combination of these additional layers sometimes creates a monster, and that’s not healthy. We may have all the computer power we ever need, but we need to think more selectively and use it more wisely.

Looking forward to the likelihood of change, I fear that market competition is fierce and we are not as candid as we should be in the way we talk about models and their limits. This is very difficult to overcome as both consultants and academia have incentives to oversell the precision (sophistication) of their tools; but the model may be very precise on a base year and very poor in forecasting years as the underlying assumptions and future inputs will be flawed. It will be interesting to be able to talk about successes and failures, failures in particular. A lot of published research is, in my view, of limited value: it is often just another small addition to layers of sophistication that merits publication rather than addressing key issues of how to handle uncertainty or a more radical review of travel behaviour. We need incentives to be more open about the difficulties we face and our limited capacity to forecast the future. I think that this is necessary in order to improve the quality of the modelling and forecasting we do. But I am not optimistic; this change will be very, very difficult to achieve.

Luis (Pilo) Willumsen has some 40 years of experience as a consultant, transport planner and researcher with a distinguished academic career. He is an internationally recognised authority in Transport and Traffic Modelling. Based in Britain since 1975, he was a researcher and lecturer at Leeds University and then at University College London. Luis was a Director of Steer Davies Gleave having joined it full-time in 1989 with a special responsibility for technical development; he left that company to set up his independent practice in December 2009. He is also a Director of Kineo Mobility Analytics specialised in exploiting big data sources. He is co-author of the “Modelling Transport” text and recently published a book on “Better Traffic and Revenue Forecasting”.