Popular social media apps use AI to analyze photos on your phone, introducing both bias and error

Digital privacy and security engineers at the University of Wisconsin-Madison found that the artificial intelligence-based systems used by TikTok and Instagram to extract personal and demographic data from user images could misclassify aspects of images. I discovered that there is a sex. This can introduce errors into age verification systems and introduce other errors and biases into platforms that use these types of systems for their digital services.

Researchers led by Qasem Fawaz, associate professor of electrical and computer engineering at the University of Wisconsin-Madison, used two We studied mobile apps on three platforms. The model accurately recognizes demographic differences and age.

The team plans to present its findings at the IEEE Symposium on Security and Privacy in San Francisco in May 2024. The findings are also available on the preprint server arXiv.

Many mobile applications use machine learning or AI systems called “vision models” to examine images on a user's phone and extract data that can be used for facial recognition or to verify a user's age. Fawaz said these models can also collect a lot of other information, such as demographic information, objects in the photo, and possible locations, but it's not clear what this data will be used for. Not so long ago, this process took place in the cloud. The vision model sends the user's data offsite to her server for processing.

“Mobile phones are fast enough now that we can actually do machine learning directly on the device. This not only saves the cost of the platform, but also uses more data and ,” said doctoral student Jack West, who worked on the project with doctoral student Shimaa Ahmed and Fawaz.

With that processing now happening on people's devices, researchers can now take a closer look at AI vision models and the types of data they collect and process.

The University of Wisconsin-Madison team analyzed the models of the two platforms to determine what information they collect and how they process it. West created a custom operating system for him to track the information input into the vision model and collect the model's output. The team did not attempt to extract or reverse engineer the vision model itself, which would violate the app's terms of service.

“I opened the app and saw where the inputs were and what the outputs were,” Fawaz explains. “We were basically observing how the app behaved.”

They found that when users select a photo to upload to TikTok from their phone's camera app, the visual model automatically predicts the age and gender of the person in the image. Using that understanding, they ran his model dataset of over 40,000 faces through a visual model and found that it made more mistakes in classifying people under the age of 18 than people over the age of 18. I understand. For people between the ages of 0 and 2, models often classified them between the ages of 12 and 12. And 18 years old.

An analysis of Instagram revealed that its vision model classifies over 500 different 'concepts', including age, gender, time of day, background images, and even the food people are eating in the photos. did.

“That's a lot of information,” says Ahmed. “He found that 11 of these concepts were related to facial features such as hair color, beard, glasses, and jewelry.”

Researchers showed an Instagram model a set of AI-generated images of people representing different ethnicities and assessed whether Instagram could correctly determine 11 face-related features of the model. Instagram was much better at categorizing images by age than his TikTok, but it had its own problems.

“It didn't perform that well across all demographics and seemed to be skewed toward certain groups,” Ahmed says.

So what exactly does the app do with this information? It's not entirely clear.

“The moment you select a photo on Instagram, whether you discard it or not, the app analyzes the photo and grows its local cache of information,” West says. “The data is stored locally on your device and there is no evidence that it was ever accessed or sent. But it's there.”

Given that Instagram and TikTok use data for purposes such as age and identity verification, researchers believe there is room for improvement in this technology. Reducing bias in this type of vision model could help ensure that all users receive fair and accurate digital services in the future, they say.

Other authors from UW-Madison include Maggie Vertig and Professor Suman Banerjee.Other authors include: Li Tiemt of the Technical University of Munich.