Trained with machine learning, this robotic feeding system is set to transform the lives of people with severe mobility issues, bringing independence to them.
A team of Cornell University researchers has developed a robotic feeding system that integrates machine learning, multi-input sensors, and computer vision to help people with severe motor disabilities eat.
Robot-assisted feeding systems are already being used to greatly improve the lives of users with limited mobility: these systems can pick up and place food so that the user can lean forward to take a bite, but not all users can lean forward.
In addition, some people who rely on these systems have limited mouth movement or opening that prevents them from using the system. Other characteristics, such as sudden muscle spasms, can also cause problems.
In these cases, users would benefit from a system that allows for precise placement of food and “mouth feeding” using an implement that can be guided by deliberate tongue movements.
A team from Cornell University has just such a system in mind, and in March they presented their robot at the Human-Robot Interaction conference in Boulder, Colorado, where it won an award for best demonstration.
“Using a robot to feed someone with severe motor impairments is challenging because many of them cannot bend over and need to put food directly into their mouth,” said senior developer Tapomayuk “Tapo” Bhattacharjee, an assistant professor of computer science in Cornell's Ann S. Bowers School of Computing and Information Sciences. “When you start feeding someone with more complex medical conditions, the challenges become even more severe.”
School lunch issues provide food for thought
In developing their robotic feeding system, the team faced a major challenge: teaching the machine the complex process of humans feeding themselves – something we often take for granted.
This involves the system identifying different foods on the plate, picking them up with the device, and then precisely transferring them into the user's mouth. Bhattacharjee noted that the most difficult part of the operation is the last two inches (five centimeters) before it reaches the user's mouth.
The system also needs to take into account the fact that some users have mouths less than an inch wide, and needs the ability to account for unexpected muscle spasms that can occur when holding the dish close or when the dish is in the user's mouth.
Additionally, the team decided it would be desirable for users to use their tongue to indicate to the system which part of their mouth they could chew food in.
“Current technology looks at a person's face only once and assumes that they are staying still, which is often not the case and can be very limiting for care recipients,” said Rajat Kumar Jenamani, lead author of the paper and a doctoral student in computer science at Cornell University.
The team's system addresses these challenges in two main ways: The feeding robot can track the user's mouth in real time, allowing it to adapt to sudden movements.
This capability is enhanced by a dynamic response mechanism, which allows the system to quickly react to changes in the physical interaction between the user's mouth and the feeding implement, allowing the system to distinguish between an intentional bite and a sudden, unintentional jerk.
Of course, with any such system, the ultimate validation is testing with human users.
The proof is in the results
The robotic element of the system takes the form of an articulated arm that can hold a custom-made instrument and sense the forces acting on it.
The system's mouth tracker was trained using thousands of images of head position and facial expressions collected by two cameras placed one below and one above the custom dinnerware, which not only detect the position of the user's mouth but also help observe any obstructions caused by the dinnerware itself.
After training the system, the team set out to demonstrate the effectiveness of the system's individual components in two separate studies, followed by a full evaluation of the system with 13 care recipients with a range of mobility challenges.
Testing was conducted at three locations: the EmPRISE lab on Cornell's Ithaca campus, a medical center in New York City and in the homes of care recipients in Connecticut.
“This is one of the largest real-world evaluations of an autonomous robot-assisted feeding system by an end user,” Bhattacharjee said. “It's fantastic and extremely challenging.”
The team noted that participants consistently emphasized the comfort and safety of the intraoral occlusion transfer system and deemed the testing of the system a success.
Test users also gave the robotic feeding system a high level of technology acceptance, the team said, highlighting the system's transformative potential in real-world scenarios. “We're enabling people to control a 20-pound robot with just their tongue,” Jenamani said.
Although these results are promising, the team needs to conduct further studies to evaluate the system's long-term usefulness.
Demonstrating the system's power to transform people's lives, Jennaman shared the raw emotion felt by the parents of a daughter with a rare congenital disorder called split-brain quadriplegia as they watched their daughter feed herself with the help of the system.
“It was a really emotional moment. Her father raised his hat in joy and her mother was close to tears,” Genamani concluded.
Reference: RK Jenamani., et al, Feel the Bite: Robot-Assisted Inside-Mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control, HRI '24: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, [2024]
Feature Image Credit: Bharath Sriraam on Unsplash
