Machine learning, a key enabler of artificial intelligence, is increasingly being used in applications in self-driving cars, medical devices, advanced robots operating in close proximity to humans, and in all situations where safety is critical.
Engineers hope machine learning will bring new capabilities and efficiencies to these systems, but it also comes with risks. For example, an autonomous vehicle could misidentify a pedestrian and cause a collision, or an autonomous robot in a factory could misidentify a worker as a moving object.
Currently, machine learning cannot be performed with the level of reliability expected in safety-critical applications, said George Pappas, associate dean for research in the University of Pennsylvania’s College of Engineering and Applied Sciences.
“There is a significant gap in machine learning performance.” [and] “In the machine learning community…people may be satisfied with 95 percent or 97 percent performance,” Pappas said. For safety-critical systems, an error of 10⁹ is desirable. In other words, a system that is error-free 99.9999999% of the time.
Pappas chaired a National Academies research committee that compiled a report on how to close that gap by improving the safety of both machine learning components and the systems in which they are embedded. He and two fellow commissioners spoke at a recent webinar discussing the new report.
Improving performance and building guardrails
Machine learning components learn by detecting patterns in training data and using that “knowledge” to inform the system’s actions and decisions. Incomplete training data or incomplete sensing of the environment can prevent the system from accurately recognizing the situations it encounters, leading to potentially harmful consequences.
One of the current problems is that training data is often collected as a side effect of normal business processes, separate from the environment in which the technology will ultimately be used, said Thomas Dieterich, professor emeritus in the Oregon State University School of Electrical Engineering and Computer Science.
Rather, data should be focused on covering the real-world environments in which the systems are expected to operate, he said. This includes data that reflects any real-world conditions encountered when the system is deployed, such as changes in weather or lighting.
Another challenge with machine learning is novelty. That is, when the system encounters something new that it could not identify in the training data. For example, self-driving cars may encounter animals or transport equipment that they have never seen before.
“Machine learning systems tend to fail when they encounter new things,” Dieterich says. “We need an outer loop that can detect and characterize novelty as it occurs, and we need a process that properly deploys both automated and human organizational processes to collect additional data and retrain and revalidate the system to ensure that discovered novelties are being handled appropriately.”
Even as data and learning processes improve, there is an inherent degree of uncertainty in machine learning evaluations, and safety-critical systems must be designed with that expectation in mind, the report says. For example, the machine learning component of a self-driving car needs to be able to indicate “high uncertainty” and the car needs to slow down, take more photos, and collect more data to learn more about the situation.
Machine learning systems can also make mistakes, such as failing to detect obstacles or overestimating the distance between one vehicle and another, Dieterich said, and system designs must include redundancies and guardrails to detect when mistakes are likely to occur and switch to more reliable backup strategies.
“Mitigating or addressing the potential shortcomings of machine learning components requires changes to the architecture of safety-critical systems,” he said.
Mr. Pappas also emphasized the need for such measures. “Every time you use machine learning in a safety-critical setting, you need to develop safety filters and guardrails,” he said. If a car or robot misclassifies someone, safety devices are needed to prevent accidents. “That’s the most pressing challenge we face,” he said.
The report also highlights the importance of new standards, regulations, and test methods to address safety challenges in these systems and protect public safety and trust.
“While there have been great advances in guidance for managing machine learning in safety-critical environments, so far these measures have not been sufficient to accomplish what we need for the future,” said Jonathan Howe, a professor at the Ford School of Engineering at the Massachusetts Institute of Technology.
Mr. Howe also emphasized the importance of transparency in both technical analysis and reporting of safety incidents, including near misses, and the need to learn from these incidents. “It’s essential to building and maintaining trust with the community,” he said.
Bridging disciplines and advancing education
Improving the reliability of safety-critical systems using machine learning means bridging disciplines and developing new scientific approaches, the report says. Today, the safety-critical systems and machine learning research and development communities differ in norms and standards, governance approaches, and cultures.
“While the machine learning community has generally focused on the average accuracy of a system, in safety-critical systems we worry about worst-case accuracy, especially in high-risk areas where we want to make sure we don’t collide with pedestrians or human workers,” Dieterich said.
Pappas points out that communities also have different design philosophies. “While the safety-critical community restricts domains to achieve specific safety performance, the goal of machine learning is to develop models that generalize across all environments.”
“Integrating these two design philosophies will require a new engineering discipline…building a new scientific community that thinks from both sides of the coin to produce principles that lead to safe products and safer engineered systems,” Pappas continued.
Howe said employee education is critical to this goal. “Educating the next generation of researchers and engineers on how to build these machine learning-enabled safety-critical systems will require intensive efforts.”
This effort should extend beyond graduation, he added. “Industry recognizes this is an important challenge and can further educate the engineers already in the field to be more aware of safety regulations and how to incorporate them into the thought process for developing machine learning capabilities.”
Related resources
