Being able to distinguish between individual animals, including their unique histories, behavioral patterns, and habits, helps scientists better understand how species function and better study habitat management and population dynamics. Currently, most computer vision systems for tracking animals are effective against species with patterns or markings, such as zebras, leopards, and giraffes. The task becomes even more complex for untagged species, where individual differences are difficult to detect. Distinguishing one brown bear from another in a non-invasive way requires an incredible eye for detail and years of observing the same bears. In addition, these bears wake up from hibernation in the spring, lose a lot of weight with their shaggy fur, and then eat salmon, gain a lot of weight, and completely shed their winter coat. This is enough to surprise experts and AI algorithms. A team of scientists from EPFL and Alaska Pacific University has developed an AI program that can recognize individual brown bears in photos over time, despite their changing appearance and the challenges associated with capturing images of the elusive, long-distance animals.
Our biological intuition was that head features combined with pose were more reliable than body shape alone, which changes dramatically with weight gain. The data proved us right – PoseSwin performed significantly better than models that used body images or ignored pose information
Machine learning based on head and posture
McNeil River State Game Refuge in Alaska is home to the largest seasonal population of brown bears in the world. Every summer, nearly 150 animals roam freely across this pristine 500 square kilometer area. They gather in high-protein sedge meadows and large low-grade waterfalls to fish for salmon, providing salmon viewing opportunities for the few humans allowed inside the reserve. “The latter is under strict surveillance. This is bear territory!” smiles Alexander Mattis, professor at EPFL’s Brain Mind Institute and Neuro-X Institute. This remote region is home to Beth Rosenberg, a researcher at Alaska Pacific University’s Institute of Fisheries and Aquatic Science and Technology, for four months of the year. She has built an incredible database of brown bear images. Between 2017 and 2022, we took more than 72,000 photos of 109 different brown bears under all conditions: in the rain, at different times of the day, in all behaviors and positions (angles), to fully depict brown bears in their natural habitat.
To develop the AI program, called PoseSwin, scientists leveraged their biological expertise to focus on four characteristics of bears that change surprisingly little over time: muzzle shape (with minimal fatty tissue), brow bone angle, and ear position. Importantly, we analyzed photos of the bear from various angles, including frontal, profile, and oblique views, and incorporated pose information. “This posed approach allowed us to use as many photos as possible, even those that didn’t show the bear’s face completely clearly,” Mathis says. “Our biological intuition was that head features combined with pose would be more reliable than body shape alone, which changes dramatically with weight gain. The data proved us right. PoseSwin performed significantly better than models that used body images or ignored pose information.”
Catch the true identity of the bear
The architecture behind the scientists’ program is based on Transformers, the same underlying technology that powers large-scale language models like ChatGPT, but has been specifically adapted for image analysis. “We used a technique called metric learning to train a transformer to understand the relationships between different parts of an image,” Mathis says. In other words, the algorithm has learned not only to recognize individual bears based on the aforementioned characteristics, but also to compare images of two bears. The researchers exposed the algorithm to a group of three photos, two of the same bear and one of a different bear, taken at different times. The algorithm projects images into a multidimensional mathematical space, placing photos of the same bear close to each other and photos of other bears farther away. “It’s a real game of attraction and repulsion, a digital tug of war in which images shuffle until they form a coherent group,” Mathis says. “Each bear ended up being represented as a unique arrangement of dots, which suggests that the AI program was able to capture something fundamental: not just what the bear looked like, but something close to its true identity.” PoseSwin can also flag bears you’ve never seen before. This is a great advantage for studies in unenclosed areas where new individuals may appear regularly.
The next step was to adapt the program to the new environment. To do that, scientists turned to citizen science. We collected photos taken by visitors to Katmai National Park and Preserve, located just over 40 miles from the McNeil River, and fed them into the PoseSwin algorithm. The program clearly identified some of the bears and specifically showed where they go foraging seasonally. “This is a concrete example of the potential of the PoseSwin model,” says Beth Rosenberg. “Ultimately, this technology could be used to analyze the thousands of photos visitors take each year and help map how brown bears use this vast area. This will help us understand what brown bears need, how brown bear population dynamics are working, and many other important ecological questions.”
“A bear is a complicated version of a mouse.”
Thanks to photos of bears and virtual measurements of their morphology, scientists can now track Sloth, Rocky, that bear, and about 100 of his friends without physically interfering. “The better we can differentiate between individual bears, the better we can understand bears and their behavior at the species level,” Rosenberg says. “Bears are at the top of the food chain, ensuring the proper functioning of ecosystems. They are essential to maintaining a healthy system.”
PoseSwin makes fieldwork more broadly applicable not only to scientists involved in research, but also to other scientists working in other settings. We also achieved excellent accuracy on the macaque monkey benchmark dataset, suggesting broad applicability beyond bears. “Bears are probably the most difficult species to individually recognize,” Mathis says. “We first focused on them because we thought we might be able to adapt the program to other species, from mice to chimpanzees, which appear to have much less variation in appearance.” The team provided open-source access to the algorithm and the data used to develop it so that other researchers could use and adapt the algorithm as they saw fit.
Scientists plan to continue developing PoseSwin for brown bears in Alaska. The program is scalable, so you can already add data collected from other seasons and other locations. Their goal is to automate much of the system so they can monitor wildlife populations over the long term.
