China's efforts to adopt AI as a means of improving access to media for deaf people is not working well, according to a professor at Beijing Normal University's Faculty of Education.
Professor Zheng Xuan wrote at an outlet called Sixth Tone and noted that China has 20 million deaf people living in China, and that the national government is providing real-time translations of several television shows, encouraging the use of avatars and virtual presenters.
Techno optimists believe these flaws will be resolved, but ethical harm should not be ignored
These efforts began at the 2022 Beijing Winter Olympics. This was not impressed by Zheng's research to assess the quality of sign language translation.
“We transcribed and translated sign language created by the avatar, compared the results to the original audio, and found that a significant amount of important information was lost or distorted in the AI-generated version,” she wrote.
The deaf people had a hard time understanding sign language generated by AI.
“In thorough examination, the movement of the avatar was very different from everyday sign language in terms of hand shape, position, orientation and movement,” she writes. “The other issues were even more pronounced. The avatar's expression and body language were turned off, and the mouth was distorted.”
In an interview, viewers reported that “in general inability to understand avatar movements and pointed out that while they struggle to process multiple meaningful words, they appear to have a limited vocabulary.”
Zheng believes that AIS did it badly because “we can't find Chinese words for the meaning expressed by 50% of gestures in Chinese sign language.”
The developer believes, “The difference between signature language and speech language. In particular, many people perceive it as an accessory for speech language or it is similar to translation between two spoken languages.” ”
“But the modalities of spoken and sign language are completely different,” she writes. “The former is oral auditory language, while the latter is visual or visual spatial language. The term “gesture” is a relatively broad concept that includes not only hand movements, but also facial expressions and body language. The complete use of the body in space allows sign language users to express the meaning of the entire sentence.
The diversity of sign language in China is also a problem. Zheng wrote that Chinese sign language comes from “natural sign language” which originates from the daily lives of the deaf person, and “signed Chinese,” a Chinese expression using signs.
“The language used by most Chinese deaf people is somewhere in the spectrum between the two,” she wrote, adding that sign language dialects are another complication for developers.
Building a translation avatars to handle all these nuances is difficult, and it becomes even more difficult due to the lack of useful data to inform AIS.
Zheng also criticized Chinese technology companies. She said, “We don't deeply engage sign language linguists and deaf people. Even when sign language teachers and interpreters are included, developers often immerse themselves in a role that supports the hearing impaired views as the ultimate arbiter of the product.”
It's not just a jab. Zheng worked as a consultant for a team working on translation avatars, but found out he was inadequately equipped for the task.
“They seemed to underestimate the difficulties involved, overestimate the power of technology to solve problems, and lacked the necessary experience, resources and capabilities to determine the quality of work performed by third-party companies,” she wrote. “By the way I joined the project, these shortcomings were already clear. The development team welcomed me into participation, but I felt that this respect was more for my technical knowledge as a university professor than my identity as a deaf person.”
When pointing out that the product doesn't satisfy users, Zheng said, “My feedback was completely unacceptable as developers seem to be unable to fully empathize with my frustration.”
She currently feels there are “basic issues regarding how Tech approaches the problem of sign language translation.” This is because “tech companies are used to launch versions with many bugs first and optimize them through a lot of user feedback.”
The products she saw were so bad that she fears they will harm the faith of the deaf community for technical solutions.
“It goes without saying that some companies use real humans rather than avatars to mislead users, and release immaturely generated AI versions,” she writes. “Techno-optimist may believe that all of these flaws will be resolved over time, but irreversible ethical harm should not be ignored. If the actual needs of the deaf person are not responded, they will feel treated as guinea pigs.” ®
