Services like Google Translate help millions of people communicate in over 100 languages. Users can type or speak words to translate. You can also translate text in photos and videos using augmented reality.
Okay, computer science professor Andrea Sarjan and Ben Guerrieri ’26 We are working on adding one more language to the list. It’s American Sign Language.
Using computer vision and machine learning, researchers are setting out to create a program that acts as a Google Translate tool for ASL speakers to sign into the camera and receive direct translations.
“We are currently looking at recognizing letters and words that have static gestures,” Sardian said, referring to the letters of the ASL alphabet without hand movement. The program initially works like a dictionary. The pair will then develop automatic translation, she explained.
Salgian’s research uses a free machine learning framework developed by Google called Mediapipe to detect joint positions in real time using cameras. The program tracks the user’s movements, provides coordinates for every joint in the hand, and uses the coordinates to extract gestures that match her ASL signature.
Computer Science Major Ben Guerrieri ’26 I discovered Salgian’s project shortly after arriving at TCNJ and am now collaborating with her on this AI research.
“This is very practical for me,” he said of his contribution to the project, which consists of researching and developing translation algorithms. “It allows us to step-by-step develop algorithms that give very compelling real-time results.”
This project is part of Salgian’s ongoing interest in and research into visual gesture recognition, including its applications in musical conducting and movement.
“ASL is a fascinating application, especially given its accessibility aspects,” says Salgian. “It makes a lot of sense to be able to communicate with people who don’t speak ASL but want to understand it,” says Salgian.
— Kaitlyn Bonomo ’23
