A Japanese research team has developed an artificial intelligence tool called YORU that can identify specific animal behaviors in real time and instantly interact with animal brain circuits. This open source software scientific progresswill allow biologists to study social interactions faster and more accurately than before. By treating complex movements as separate visual objects, the system allows computers to “watch” behaviors such as courtship and food sharing and respond within milliseconds.
Biologists have long struggled to automate the analysis of how animals interact. Social behaviors such as courtship and aggression involve dynamic movements, and individuals often come into contact with each other or hide each other from camera view. Previous software solutions typically relied on a method called pose estimation. The technology tracks specific points on the body, such as joints, knees, and wingtips, over many video frames and calculates their movements.
These old methods often fail if the animals get too close to each other. When two insects overlap, the computer often loses track of which leg belongs to which individual. This confusion makes it difficult to start experiments at the exact moment the behavior occurs. To solve this, a team including Hayato Yamauchi and Ryosuke Takeuchi sought a different approach. They conducted their research under the guidance of senior author Azusa Kamikawachi from Nagoya University.
The group aimed to create a system capable of “closed-loop” feedback. This term refers to an experimental setup in which a computer observes the animal and instantly generates stimuli in response. For example, a computer might turn on a light the moment a fly stretches its wings. Achieving this requires software that processes video data faster than the animal’s movements.
The researchers built the system using a deep learning algorithm known as object detection. Unlike pose estimation, this method analyzes the entire shape of the animal within a single video frame. The team named the software “YORU.” This acronym stands for Your Optimal Recognition Utility.
YORU identifies specific actions as separate “action objects”. The software recognizes visual patterns such as two ants sharing food or a male fly vibrating its wings. This approach allows computers to classify social interactions even when animals are in contact. By viewing motion as a unified object rather than a collection of points, the system avoids the confusion caused by overlapping limbs.
The research team tested YORU in several different species to verify its versatility. They videotaped the fruit flies’ courtship, the ants’ mouth-to-mouth transfer of food (a behavior known as trophalaxis), and the zebrafish orienting themselves to each other. The system achieved detection accuracy in the range of approximately 90-98% compared to human observation.
This software also proved effective in analyzing brain activity in mice. The researchers placed a mouse on a treadmill within a virtual reality setting. YORU accurately identified movements such as running, grooming, and whisker movements. The system matched these physical movements with simultaneous recordings of neural activity in the mouse cortex. This confirms that AI can reliably link visible movement with invisible neuron firing.
The most advanced tests involved a technology called optogenetics. This method allows scientists to use light to switch specific neurons on or off. The research team genetically engineered the neurons responsible for courtship songs in male fruit flies to be silenced by green light. These neurons are known as pIP10 descending neurons.
YORU was watching flies in real time. When the system detected a singing male with wings outstretched, a green light turned on within milliseconds. The male fly immediately stopped its courtship song. This disruption caused a statistically significant decrease in mating success.
Co-lead author Hayato Yamauchi from the Nagoya University Graduate School of Science emphasized the differences between the two approaches. He said, “Rather than tracking points on the body over time, YORU recognizes entire behavior from its appearance within a single video frame. YORU detected behavior in flies, ants, and zebrafish with 90-98% accuracy and performed 30% faster than competing tools.”
The researchers then took the experiment a step further by using a projector. They wanted to manipulate only one of the pair without affecting the other. They genetically modified female flies to have auditory neurons that are sensitive to light. Specifically, they targeted neurons in Johnston’s organ, which corresponds to the fly’s ear.
Once the male fly extended its wings, YORU calculated the exact location of the female. The system then projected a small circle of light onto her chest. This light silenced her auditory neurons at the exact moment the male tried to sing. The female ignored the male’s complaints because she could not hear him.
This experiment confirmed the software’s ability to target individuals within a group. Azusa Kamikawachi explained the importance of this accuracy. “We can silence the fly’s courtship neurons the moment YORU senses wing extension. In another experiment, we used a targeted light that tracked individual flies, blocking just one fly’s auditory neurons while the other flies moved freely nearby.”
The speed of the system was the main focus for the researchers. They benchmarked YORU against SLEAP, a popular pose estimation tool. YORU’s average latency (delay between seeing an action and reacting to it) was approximately 31 ms. This was approximately 30% faster than the alternative method. Such speed is necessary to study neural circuits that operate on very fast timescales.
The system is designed to be easy to use, even for biologists who are not experts in computer programming. It includes a graphical user interface that allows researchers to label behaviors and train AI without writing code. The team made the software open source, allowing labs around the world to download it and adapt it to their own specific animal models.
This system provides speed and accuracy, but relies on the appearance of motion within a single frame. This design means that YORU cannot easily identify behaviors that depend on a sequence of events over time. For example, additional analyzes may be required to distinguish between the beginning and end of foraging behavior. The software excels at identifying “states” of existence rather than complex narratives.
Additionally, the current version does not automatically track the identity of individual animals over time. If two animals look the same but switch places, the software may not be able to tell them apart without an auxiliary tool. Researchers may need to combine YORU with other tracking software for studies that require long-term personal medical histories.
Hardware limitations present another challenge for projector-based systems. If there is a slight delay in the projector, fast-moving animals may exit the illumination area before the light pulses. Future updates may include predictive algorithms that predict where animals will be every millisecond.
Despite these limitations, YORU represents a new way to interrogate the brain. By allowing computers to recognize social behavior on the fly, scientists are now able to ask questions about how the brain navigates the complex social world. The ability to turn on or off certain senses during social interactions opens new avenues for understanding the neural basis of communication.
The study, “YORU: Animal behavior detection using an object-based approach for real-time closed-loop feedback,” was authored by Hayato Yamauchi, Ryosuke Takeuchi, Naoya Chiba, Koichi Hashimoto, Takashi Shimizu, Fumitaka Osakata, Yoshiya Tanaka, and Azusa Kamikawachi.
