The automotive industry faces a monumental challenge. When it comes to vehicle testing and verification, previous encounters do not approach the complexity of autonomous vehicle technology.
For a vehicle to complete its journey autonomously, it must be able to negotiate a wide range of scenarios captured within a wide range of operational design domains (odd numbers). Repeating any possible combination of these factors is simply not practical. Even capturing those reasonable cross-sections can be very expensive and time-consuming with traditional testing methods.
This raises some basic questions. How many tests do self-driving cars need before they are considered safe? How do you choose the best value scenario to rate? And how can you optimize the time and resources needed to do this without compromising safety?
A new approach to vehicle testing
In 2023, Horiba Mira partnered with Polestar, IPG Automotive, The Connected Places Catapult and Coventry University to develop a toolchain that speeds up the deployment of autonomous driving and advanced driver assistance systems (ADAS) technology. The project, known as Certus, proposed a fundamentally different approach to vehicle testing. Instead of evaluating millions of different scenarios, we ingest structured test data to identify trends to determine where to prioritize the next set of test cases.
For OEMs and technology developers, the main purpose of CERTUS is to reduce the time and cost involved in both developing automated driving systems and verifying the tests of final products before release. To accelerate the final release of these systems, CERTU provides a measure of residual risk. Quantify reliability on system performance and allow developers to measure what needs to be further tested to reach acceptable levels of risk before deployment.
Machine learning and drive scenario choices
The key question of certificates is, how can I maximize the usefulness of a certain amount of testing? The traditional approach is to generate scenario spread across odd numbers using statistical techniques such as Latin hypercube sampling. As a complementary approach that OEMS can accept to identify coverage gaps in existing data, Certus provides efficiency by identifying where high levels of uncertainty and generation scenarios exist in these areas. This supports drives to ultimately reduce testing and verification spending without compromising safety.
When performing the two comparisons, using the Latin Hypercube model, randomly generating 2,500 scenarios across the problem space returned a 56% confidence rating.
The machine learning algorithm began with a similar approach using Latin hypercube samples, selecting the first 100 scenarios that provided initial indications of system performance. Nine iterations of 50 scenarios were then followed, with the algorithm identifying the region of maximum uncertainty across each spread, focusing on the next iteration around those points.
This helps you direct your test resources to the most important areas and avoid unnecessary repetition if you are already highly confident. In total, we ran 550 scenarios (five times less than the traditional approach, but ultimately got a 77% confidence rating.
This showed a significant increase in efficiency. We are also pleased that the toolchain ensures significant efficiency in many other areas. Yes, some human input is still needed, but that manual effort is reduced by certificates throughout the testing and verification process. That is, by combining the use of data analysis tools with other aspects of the Certus toolchain, including residual risk measurements, with additional insights provided by the tools developed to measure system performance. Together, this allows the scenario to achieve a five-fold reduction, ensuring a 40% reduction in overall testing time.
Oracles: Overall view of system performance
Addressing the broader challenges of system performance is not trivial, but there are several ways to quantify how well your system is working. Traditionally, the industry has used a variety of key performance indicators (KPIs). One of these is “Clash Time” (TTC). This estimates the time it takes for two vehicles to collide if both speeds and headings remain constant.
Each of these KPIs provides useful performance metrics, but when viewed alone, they do not provide a complete image. For example, if the system's response to a scenario leads to a collision, it will automatically fail with some of the key criteria. However, a conflict is inevitable and could have been carried out just as the system was executed. Conversely, the system may satisfy the metric, but it becomes another safety issue. For example, the applied braking force can be excessive. Fills the metric to stop before the obstacle, but it can trigger a rear-end collision in the process.
To provide a more comprehensive assessment, we have developed a tool known as Oracle. This not only combines multiple KPIs to provide an overall view of the performance of a system, but also provides a score that objectively evaluates its performance. Oracle is developed for safety, progression and residual risk.
Ensuring vehicles can move safely and achieve good progress has always been a major objective in the research we undertake through CERTU, and in that respect our progress and safety oracles are built to complement each other.
While the TTC KPI is central to achieving safety objectives, there are also a variety of KPIs for progression, which are combined with their assessments to generate an overall system performance score from oracle in a particular scenario.
An example of a progress KPI is the progress rate (PR). This evaluates that the vehicle is either moving towards the goal before the end of the scenario or based on mission time efficiency (MTE).
Quantification of residual risk and its benefits
These methods allow you to determine the performance of your vehicle. However, we only provide snapshots based on the results of a specific test or simulation. You can't test all of the billions of different scenarios a vehicle might encounter. So, the next question is that we are confident that the behavior of the vehicle in that particular test is representative of general performance in such a type of scenario.
Our approach is primarily based on predictability. If two closely related scenarios return very different results, it suggests that the system may behave irregularly in one or both of these situations. Conversely, similar results suggest that the system can respond consistently and proportionally to the range of different scenarios.
Repeating this process at various points throughout the scenario space allows you to measure where the vehicle is running with a high degree of confidence and where further work may be needed. For example, if there is greater uncertainty at high speeds, vehicle manufacturers may choose to carry out further developments in this area or simply limit vehicle speeds for this feature.
Quantifying this risk is useful as it indicates the distance that the test coverage truly extends. It's not uncommon to hear vehicle developers talking about “multiple million miles of testing,” but it doesn't guarantee that a wide range of different scenarios are being evaluated.
Test efficiency is key in a world where vehicle complexity is rapidly increasing and time to market is rapidly shrinking. While Adas Technology is already common, OEMs understand the pressure to perform when it comes to bringing new technology to their showrooms faster than their rivals.
As a race to acquire autonomous technology, tools such as Certus offer a strategic advantage by enabling developers to increase testing efficiency and manage risk.
