Identifying statistically robust methods to evaluate road safety projects and programs for Victoria's Department of Transport.
In November 2020, Abley’s Road Safety team began working with the Victorian Department of Transport’s Safer Roads team to identify suitable statistically robust methods to evaluate road safety projects and programs.
Victoria’s Safer Roads Program exists to eliminate road fatalities and serious injuries through speed management and road safety infrastructure improvements.
Victoria has set a goal of a 50% reduction in crashes leading to fatalities by 2030. Understanding the effectiveness of past road safety treatments is central to identifying high-priority, high-impact infrastructure improvements for future programs.
We’re sharing learnings from this project to help road safety teams better evaluate your past efforts, accurately identify which projects get the best results, and focus future funding for a greater reduction in deaths and injuries.
Bringing together a global team of road safety experts
Amir Sobhani is the Insights and Evaluation Lead for Victoria’s Safer Roads Program.
He explains why the Department of Transport engaged Abley to develop a Safer Roads evaluation framework for Victoria. “Abley’s team of road safety experts would be one of the best in the world. I knew Shane Turner would be able to put together a fantastic team who would handle this project efficiently and effectively. And Abley are very well connected. Shane brought in Bhagwant Persaud, who’s a leading expert in road safety in North America, and well-known statistician Graham Wood.
Our Safer Roads team is in charge of delivering the insights and evaluation that inform our Safer Roads strategy. We’re spending around $1.79 billion in order to improve the safety of the road infrastructure in Victoria and this investment must be evidence-based. We need to evaluate the effectiveness of all the development and delivery we do as a part of our Safer Roads Program.
First, we needed to clearly understand our business requirements for the Safer Roads evaluation framework. This required consultation with key stakeholders. Next we needed to identify the best road safety evaluation methodology to deliver these requirements. That’s not a simple thing to achieve because many road safety assessment methodologies have been developed in the last 50 years. Next, we identified the data we need to collect to start using our new methodology. And finally we developed a road map for implanting our new evaluation framework.”
Developing the evaluation framework
Developing Victoria’s road safety evaluation framework was a six-step process.
- Review previous reports and analysis to identify any gaps and potential improvements that could be applied to the Safer Roads program evaluation.
- Conduct a workshop and series of interviews with key stakeholders to identify their business requirements, practices, and insights for the road safety evaluation framework, and consider the gaps identified.
- Review road safety assessment methodologies, identify the most statistically robust methods to select future projects and programs, and propose the most practical methods for Victoria. Develop a framework to systematically allocate identified methods against identified gaps, and prioritise short, mid, and long-term solutions.
- Conduct a second workshop with stakeholders to present progress. Summarise existing road safety methodologies, present recommended methodologies for a new evaluation framework, and obtain an initial agreement to proceed on this basis.
- Recommend short to mid-term improvement activities to implement the new evaluation framework and develop a roadmap for all activities.
- Document our findings and report our recommendations to the Safer Roads team. The report included:
- Gap analysis of previous program evaluations
- Business requirements for future evaluations
- Evaluation methodology options
- Preferred methodologies
- Data requirements
- A roadmap to collect data and begin implementing new methodologies to assess effectiveness of road safety projects and programs
The new evaluation framework outlined in the report has been approved by the working group and other stakeholders. Amir’s team have now started implementing the plan.
Amir explains,
There are three phases for this project. The first phase, from now until 2022, is pulling together our existing data in a central database. That’s a big project. In the second phase we’ll establish our evaluation process using the methodologies we’ve adopted. Also, we’ll be collecting more data and continually improving our database. The third phase is about enriching our data and refining our methodology, so we get better and better at assessing the effectiveness of our road safety projects.”
Common deficiencies in road safety assessment methodology
Abley project lead Shane Turner outlines three common over-simplifications Abley comes across when assessing road safety programs.
- Regression to the mean. High risk sites tend to see fewer crashes over time, even without safety interventions. It’s important to allow for this reduction when assessing the effectiveness of road safety treatments.
- Not all reductions in FSI are due to infrastructure improvements. We’re seeing a reduction in crashes over time due to safer vehicles and more effective enforcement. Account for this underlying trend, when assessing improvements.
- Changes in road user numbers. If the number of road users increase, you may see a commensurate increase in crashes, and vice versa.
If these nuances are not effectively assessed and factored into your calculations, you run the risk of overestimating or underestimating the impact of your road safety program.
Advanced road safety assessment methods
Shane explains how Abley approaches road safety assessment methodology.
In Victoria we used best practice and techniques developed in North America. We’re on working groups there, so we’ve a good understanding of road safety evaluation methodologies developed in the US and Canada. At the same time, we’re promoting progressive Safe System thinking from New Zealand and Australia to North America. We’re cross-pollinating the best road safety practice from across the world.”
Abley recommended Victoria adopt new safety science assessment methodologies.
Empirical Bayes. The Empirical Bayes evaluation method is considered superior to other before and after crash analysis methods, including those that use control sites. Using crash prediction models (also called safety performance function in North America) accounts for the three key causes of error in road safety program evaluations, being regression to the mean, the trend towards fewer crashes over time, and changes in road user numbers. This approach is outlined in the USA Highway Safety Manual, which contains state -of-the-art practices in road safety analysis.
Predictive models. At Abley we favour proactive crash predictive models over historical data, because it’s a better way to understand the underlying crash risk at a location. If you’re looking to reduce fatalities across your network it’s important to know where the greatest risks are of recurring severe and fatal crashes, not simply where there’s been a flare up of crashes in the past. Crashes can appear like lightning strikes, so relying on historic crash data can be like playing whack a mole.
Balancing many business requirements
Amir explains, “The most challenging part of this project was always going to be satisfying all our stakeholders’ business requirements.
“The project working group was representative of road safety experts from the Department of Transport, the Transport Accident Commission, Metro Surface Transport, Safe System Design, Road Safety Victoria, and Regional Roads Victoria, as well as the opinion of senior leaders and directors. So, it was a combination of people with technical road safety expertise, and people with little technical background.
“All the requirements outlined by the working group were very important, but some requests were technically challenging, beyond our resource, or out of our scope. And because of the varying levels of expertise on the working group, balancing stakeholders’ business requirements required careful consideration and education.
“But I was quite confident Abley could achieve the objectives we were after. Their ability to deliver became quite obvious in their first workshop with our working group, where the Abley team inspired confidence in everyone involved.
“When we started this project, our ideal outcome was to review all current road safety methodologies, talk to all the relevant road safety experts in Australia, and recommend the right methodology for Victoria to assess road safety projects. We needed a high-level roadmap outlining implementation phases, and a timeline for each phase, and we needed to do it all in four months. It was a very short project and there was a lot to review, but Abley achieved our goals on time and on budget.”
Abley’s road safety expertise is exceptional. Evaluating road safety projects is complex. Having their analytical minds to consider all the dimensions helped us a lot. And their international lens was invaluable. To have so many road safety experts on one project is very rare.”
Amir Sobhani, Insights and Evaluation Lead,
Victoria’s Safer Roads Program
An unprecedented road safety review
Amir has advice for transport agencies developing a road safety evaluation framework. “A road safety evaluation playbook is a must from my point of view, because without a robust evaluation framework, no major road safety evaluation project will be successful.
Many people have been following this evaluation program because, to my knowledge, no one has done this detailed a road safety methodology review before. The Victorian Safer Roads Program is unique in terms of how we’re going to evaluate our $1.79 billion expenditure. Which is why my team are working with Abley to document this project in a paper we plan to present at the next road safety conference in Melbourne and publish in road safety journals.
“However, our methodology is for Victoria. If you’re adopting an evaluation framework for your country or state, don’t simply repeat our approach. What works here may not apply to you. For example, we’ve gone for an empirical base. This might not be doable for you because your business requirements, your data, and your team’s expertise will all be different. So, get experts like Abley to review your process.
Abley’s road safety expertise is exceptional. Evaluating road safety projects is complex. Having their analytical minds to consider all the dimensions helped us a lot. And their international lens was invaluable. To have so many road safety experts on one project is very rare. Organisations in Australia struggle to find this expertise, so I recommend Abley highly if you’re developing a road safety evaluation framework.”
