The path to better forecasts - keep an open mind and go with your gut

Tom van Vuren explores why experts should be prepared to challenge their own preconceptions when forecasting, embrace uncertainty and be encouraged to update their estimates when faced with new information

We transport modellers are not in the business of predicting the future. That’s not what we do. No, our job is to forecast the overall and combined impacts of certain interventions (infrastructure projects, policies) against a set of assumptions about the social, demographic and economic development of the area we assess, and the people who live there.

That is not to dismiss the points about “how not to predict the future” made by data analyst Molly Hickman in the online Asterisk magazine. On the contrary, her piece offers a good set of anchors for both advocates and sceptics. We can improve practice and practical use, by embracing  Hickman’s arguments.

According to her research, the best forecasters have a quality sometimes described as “active open-mindedness”. Instead of operating on autopilot, those with an active open mind use “evidence that goes against their beliefs, pay attention to those who disagree with them, and are willing - even eager - to change their minds”. 

This is not necessarily very easy for subject matter experts who are heavily invested in particular narratives, Hickman explains. This, she adds, applies just as much to those that believe all models are right and those that believe all models are wrong.

It is no less of an issue in the discussion about open source vs commercial off-the-shelf models. Leaving monetary interests aside, many forecasters are probably too invested in the modelling tools they use. This makes it difficult to actively progress the discussion around what really matters. Should it be better forecasts, more inclusive access to forecasts or a more helpful role of forecasts in decision-making? See also my recent TransportXtra interview with Robin Lovelace.

Experienced forecasters should use quantitative models, Hickman believes. “For complicated questions, this usually involves picking a decomposition - that is, a way of breaking down a hard problem into smaller, more manageable pieces,” she states. “Decomposing questions is a major part of forecasting, and when it’s done well, it can help ground our intuitions and keep our biases in check.”

And that’s true for transport models, too. Active open-mindedness challenges the assumptions made, and components that are included or excluded in that decomposition.

Can we assume that active modes can be safely ignored? Is the study area unnecessarily large (or small)? Is a highway-only model the right tool? What would be the effect of a policy or project if we allow land use development to respond to accessibility changes? Do we need to focus on activities rather than travel movements? 

Hickman continues: “Good arguments about forecasts often revolve around decompositions. Different decompositions can yield radically different results - so try a few and don’t be precious. Average them, give more weight to the ones that make more sense to you (an ensemble model) … and then go with your gut.”

Go with your gut? Absolutely! Rather than treating any quantitative model or forecast as the only truth, they and their assumptions can be (should be!) used to explore and explain (or explain and explore, you choose).

To further quote Hickman: “It’s almost always worth the effort to make a quantitative model - not because its results are the immutable truth but because practising decomposing questions and generating specific probabilities are how you train yourself to become a better forecaster.”

Mistakes that professionals make when forecasting, are according to Hickman, two-fold. “The first mistake is in trusting our preconceptions too much. The more we know - and the more confident we are in our knowledge - the easier it is to dismiss information that doesn’t conform to the opinions we already have”. Experts make for terrible forecasters, Hickman says.

And I regularly experience that too; whenever the forecast does not confirm with an expert’s worldview, the model must surely be wrong. Having said that, Hickman also mentions a second kind of error: “Putting too much store in clever models that minimize the role of judgment. Just because there’s math doesn’t make it right.” She goes on: “You might find yourself thinking: This model is so fancy, it must be right… but 10% just feels too high!” 

So, the trick is active open-mindedness. I feel compelled to finish by paraphrasing my own recent TransportXtra article on the importance of language when communicating forecasts, to back Hickman’s points up.

Forecasts needs to be near enough to be reliable, but at the same time, we need to know about their uncertainty. You can guide people by saying that it is an estimate, there’s no guarantee that this is going to exactly reflect the real world. And the more you can do to put some sort of numerical context around that, the more reliable basis you have for people who are using those numbers.

And remember - forecasts must inevitably change over time when new information becomes available, and when assumptions get adjusted in response. This, of course, does not mean that forecasting is pointless – but throughout the use of numbers that transport models produce, uncertainty and the possibility of updating them to more up-to-date values, must be recognised and communicated, as a positive process rather than an excuse.

Tom van Vuren is Chairman of Modelling World, Strategic Consulting Partner at Amey, and a Visiting Professor at the Institute for Transport Studies at the University of Leeds

Modelling World 2024 https://www.modelling-world.com