Tackling uncertainty: superforecasting comes to Modelling World 2020

It turns out that the Superforecasters were quickly able to develop early, accurate forecasts on COVID-19. Time Magazine actually called them ‘eerily accurate’ in a piece back in June, and they have consistently outperformed epidemiologists and biostatisticians from around the world in predicting the course of the pandemic

Back in 2011, the Good Judgment Project, led by Philip Tetlock, author of Superforecasting: The Art and Science of Prediction, was the undisputed victor in a tournament led by the US-based Intelligence Advanced Research Projects Activity (IARPA) to gauge the accuracy of forecasts

Back in 2011, the Good Judgment Project, led by Philip Tetlock, author of Superforecasting: The Art and Science of Prediction, was the undisputed victor in a tournament led by the US-based Intelligence Advanced Research Projects Activity (IARPA) to gauge the accuracy of forecasts

Warren Hatch, CEO, Good Judgment

Warren Hatch, CEO, Good Judgment

Warren Hatch, CEO, Good Judgment, will be joining us at Modelling World 2020 on October 16, virtually, to share his skills, insights and experience and to engage with delegates in practical demonstrations. Back in 2011, the Good Judgment Project, led by Philip Tetlock, author of Superforecasting: The Art and Science of Prediction, was the undisputed victor in a tournament led by the US-based Intelligence Advanced Research Projects Activity (IARPA) to gauge the accuracy of forecasts. Good Judgment’s forecasts were so accurate that they outperformed intelligence analysts with access to classified data, said IARPA. 

Speaking to Forbes late in 2019, Warren Hatch said: ‘Good Judgment Inc is the commercial successor to the research project that was led by Philip Tetlock at the University of Pennsylvania and the University of California, Berkeley. The work began with the US government wanting to see if there were ways to improve forecast accuracy beyond the so-called “wisdom of the crowd”.  I had signed up for the research project and started emulating the behaviours of the best forecasters.’ After about 20 questions, there was a click, said Hatch, and he got his “aha” moment. He became a superforecaster.

Good Judgement now has a network of superforecasters working on economic, social and investment issues, including transport. Hatch holds a doctorate in politics from Oxford, a masters in Russian and international policy studies from Middlebury Institute of International Studies at Monterey, and a bachelors in history from the University of Utah. 

The epidemiologists hope that such ‘ensemble modeling' will generate better outputs for global health organisations. So while COVID-19 has shaken everything up, best-methods forecasting is able to quantify a lot of that additional uncertainty

Speaking to TransportXtra last week about increased levels of uncertainty across all aspects of life since the pandemic took hold, Hatch was cautiously confident. ‘There’s more uncertainty across the board. But Superforecasters are fine with that, and have quickly adjusted their thinking. Overall, we did not know beforehand whether their skills—honed for forecasting geopolitics and geoeconomics—would transfer to forecasting a pandemic.

'It turns out that the Superforecasters were quickly able to develop early, accurate forecasts on COVID-19. Time Magazine actually called them ‘eerily accurate’ in a piece back in June, and they have consistently outperformed epidemiologists and biostatisticians from around the world in predicting the course of the pandemic. We are working now with several epidemiologists on ways to combine forecasts from their models with the consistently accurate forecasts from our humans.

'The epidemiologists hope that such ‘ensemble modeling will generate better outputs for global health organisations. So while COVID-19 has shaken everything up, best-methods forecasting is able to quantify a lot of that additional uncertainty.'

The superforecaster and the modeller

Tom van Vuren, Chairman of Modelling World and an experienced modeller who has had an eye on superforefcaster for many years, is looking forward to meeting up with Hatch at this year’s event.

He says: ‘I wrote in 2016 about superforecasting, and am super-excited to have a real superforecaster present at Modelling World this year. My 2016 thinking was influenced by the Freakonomics Radio Podcast: 'How to be less terrible at predicting the future'. Although the predictions discussed in the podcast are much more related to politics, economics or sports than to transport policy or infrastructure demand forecasting, our own profession can also benefit.

People are bad at forecasting because there is no danger in getting it wrong. That is sometimes true for us modellers as well. Even if our forecasts look wrong, we present ourselves as being powerless, as the future growth assumptions are prescribed by third parties including Government (even though our models are correct...).  But we can do better – we were not asked to produce (just) a robust model, we were also expected to provide a decent forecast. Challenging the inputs should be part of what we do – who is better placed than us? A bad forecaster is systematically over-confident and often dogmatic in his/her view of the world, unwilling to change his/her mind in response to new evidence. Good forecasters are humble about their judgments, are curious and are willing to challenge conventional wisdom. 

The average forecast derived by a group of forecasters tends to be more accurate than most of the individual forecasts from which the average was derived; we are more likely to provide better forecasts if we are willing to learn from other experts. That’s what Modelling World is all about so get ready to hear van Vuren and Hatch in conversation at Modelling World!