If we want safer roads, we need higher standards in casualty data analysis

Mike Maher is an honorary professor at University College London and has carried out research on the mathematical and statistical modelling of transport problems for over 40 years, specialising in network modelling and accident data. He is a fellow of the Institute of Mathematics and its Applications, and a Chartered Statistician.

Mike Maher is an honorary professor at University College London and has carried out research on the mathematical and statistical modelling of transport problems for over 40 years, specialising in network modelling and accident data. He is a fellow of the Institute of Mathematics and its Applications, and a Chartered Statistician.

Anyone looking for a simple answer to the question “Do speed cameras save lives?” would have been disappointed and confused by the widely differing views and information presented in the media in recent years.

In September 2014 a TfL press release stated: “Safety cameras have proved successful in reducing road casualties in recent years. At locations where safety cameras operate in the capital, research shows that the number of people killed or seriously injured fell by an average of 58 per cent”. What it did not say was that this percentage fall in KSIs at camera sites matched that across London generally over the same period: that is, no allowance was being made for trend.  TfL is not alone in making such an error: previously, the Scottish and Welsh camera partnerships have, on their websites or in annual bulletins, made similar misleading claims about the benefits of speed cameras in reducing casualties. It was curious that TfL’s managing director of surface transport, in rebutting criticism of the performance of London’s cameras, used evidence from the four-year evaluation report of the National Safety Camera Programme in 2005 rather than that in the analysis on the London camera data I discussed with and provided to TfL staff in October 2014.   

Another mistake commonly made is that collision rates after installation are compared with a “baseline” rate where the baseline is the period immediately before installation. Such a comparison will almost certainly be contaminated by the effect of regression to the mean (RTM), or “bias by selection”, and lead to an exaggerated estimated of the camera benefit. The phenomenon of RTM has been known for over a century, with Galton noting in 1886 that “tall fathers tend to have sons that are also tall but not as tall as themselves”.  

In traffic safety analysis, the relevance of RTM has been known since the early 1980s. So it is disappointing that many road safety professionals are seemingly unaware of the need to allow for trend and RTM when estimating the effectiveness of any remedial treatment, including speed cameras. RTM arises whenever there is a mix of systematic and random variation and selection for treatment is made on the basis of a high observed value. After selection, the systematic component persists but the random one does not, so the observed value will tend to fall. In lectures I used the example of championship golf scores. Golf is a game of skill but luck also plays its part, as any golfer will confirm. If you examine the scores of, say, the leading ten players after two rounds of the Open, and then compare their scores in rounds 3 and 4 with those in rounds 1 and 2, you will find they are, on average, around four strokes per round worse. There is no reason for the good luck that contributed to them being at the top of the leaderboard after two rounds to persist over the next two rounds. So, if sites are selected because of a large number of accidents in the last three years, it is highly likely that even without treatment there would be fewer accidents in the next three years. 

It is therefore necessary to either allow for or avoid RTM. Appendix H of the four-year evaluation report on the National Safety Camera Programme applied a technique known as the empirical Bayes method to a small (and not necessarily representative) subset of sites and showed that the size of the camera effect was reduced appreciably from 50% to 20% for fatal and serious collisions after allowing for RTM. However, although the method is widely used in traffic safety modelling research, it is often seen by non-statisticians as complex and obscure. 

The alternative “four time periods” method used by Dave Finney and then Richard Allsop in his RAC Foundation report is undeniably simpler to understand.  The data for each camera is split into a before period, a site selection period (SSP), an after selection but before camera (ASBiC) period and an after period.  The SSP contains the abnormally high collision rates so, once identified, can effectively be removed so as to permit a comparison of the rates in the before and after periods to obtain an estimate of the reduction due to the camera. 

A serious problem though is that although typically the installation date is known, the SSP is not, and some assumptions need to be made in order to apply the method. If the assumed SSP is not correct, some part of the RTM effect may be contained in the before period leading to an exaggerated estimate of the camera benefit. Allsop recommended going back four years before installation but there is no guarantee that this will be appropriate for all data sets. Indeed, in my analysis of the London data, I found that it was necessary to go back five years. The modelling then showed no significant camera effect in either the personal injury collisions (PICs, or slight injuries) or the FSCs (fatal and serious collisions). My modelling of the data from the ten partnerships covered in the RAC Foundation report produced estimates of average camera benefit of 15% in PICs and 25% in FSCs: very similar to those of Allsop.

The analysis of camera data is far from straightforward, largely because of the highly random nature of accident frequencies and the imperfect nature of the data available. The estimated benefits of cameras vary from one partnership to another. Unfortunately, the quality of the debate is often poor, bedevilled by a lack of awareness of critical issues such as trend and regression to the mean, and not helped by misleading information on websites and in press releases.