AI-generated responses undermines crowdsourced research studies

Online surveys are overwhelmed by AI-generated responses. This can contaminate important data sources aimed at scientists.

Platforms like prolific wage participants have small amounts to answer questions raised by researchers. They are popular among academics as an easy way to attract participants for behavioral research.

Anne-Marie Nussberger and colleagues at the Max Planck Human Development Institute in Berlin, Germany decided to investigate the frequency with which respondents use artificial intelligence after they noticed examples from their research. “The incidence we were observing was really shocking,” she says.

They believe that 45% of participants who pasted a single open-ended question into a box about prolific copied content indicates that people put their questions in AI chatbots to save time.

Further investigation into the content of the responses suggested that they spoke more obviously about AI use, such as “overly redundant” and “obviously non-human” languages. “From the data we collected earlier this year, it appears that a significant proportion of research is contaminated,” she says.

In subsequent studies using Prolific, researchers added traps designed to snare them using chatbots. Two Recaptchas – a small pattern-based test designed to distinguish humans from bots – caught 0.2% of participants. More sophisticated Recaptcha, using information about user past activity and current behavior, removed an additional 2.7% of participants. Although invisible to humans, bots who ask the word “hazelnut” to be included in their reactions received an additional 1.6%, while an additional 4.7% of people were identified when they prevented copying and pasting.

“All we need to do is respond and respond to online surveys rather than completely distrust,” says Nussberger. It's the researcher's responsibility, she says, and they should deal with answers with more suspicion and take steps to stop AI-responsive behavior. “But what's really important is, I think the platform has a lot of responsibility. They need to take this issue very seriously.”

In a statement New ScientistA Prolific spokesman said: “Prolific has been tackling the challenges of AI through online research for over a year and has developed solutions to protect the quality of your data.

“The integrity of online behavioral research has already been challenged by participants in research sites that use bots to misrepresent themselves or use bots to acquire cash and vouchers, not to mention the effectiveness of remote self-report responses to understanding complex human psychology and behavior.” “Researchers need to either collectively consider ways to remotely validate human involvement or return to the old-fashioned approach of in-person contact.”