According to three researchers, machine learning models can do the work of content processing and data sanitation better and more affordable than those who participate in crowdsourcing platforms.
That’s not necessarily a bad thing for job seekers, as some of the jobs that are likely to be affected look pretty awful.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli, researchers at the University of Zurich, examined how OpenAI’s large-scale language model ChatGPT processed text annotations. We compared this to Amazon Mechanical Turk (MTurk), a crowdsourcing platform, by adding labels to text so that machine learning models can better understand it.
The researchers describe their findings in a paper entitled “ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks” whose title serves as a spoiler.
Using a sample data set of 2,382 Twitter posts that had already been labeled by research assistants, Boffin compared how ChatGPT and MTurk workers performed five different labeling tasks.
This work included assessing how each tweet related to the content moderation subject in terms of relevance, stance, topic, and issue framework (e.g., content whether moderation is described as a problem that restricts speech or as a solution to protect against harmful speech).
“We found that ChatGPT has better zero-shot accuracy than MTurk for four out of five tasks,” the paper states. “…Furthermore, ChatGPT is significantly cheaper than his MTurk. Five classification tasks cost approximately $68 (25,264 annotations) for ChatGPT and $657 (12,632 annotations) for MTurk.”
For each annotation, ChatGPT costs about $0.003, roughly 20 times cheaper and more accurate than MTurk, the researchers said.
More accurate is less accurate in this case. Fabrizio Girardi, Professor of Policy Analysis at the University of Zurich’s Department of Political Science and one of the paper’s co-authors, said: register ChatGPT’s results were less than 50% accurate on some tasks, but still outperformed MTurkers.
Overall, the study results seem to be game over for human workers eager to secure this type of job.
But Gilardi warned against reading the findings too broadly.
“It is too early to say how ChatGPT will replace cloud workers.” register“Our paper demonstrates ChatGPT’s potential for data annotation tasks, but more research is needed to fully understand ChatGPT’s capabilities in this area.”
Gilardi said it is important to collect more data using different tasks, data types and languages, and MTurker has been used in research studies, image annotation, audio and video transcription, usability testing, etc. He added that he is doing other work for There may also be scenarios where human annotators are more productive with the help of models like ChatGPT, he suggested.
According to Gilardi, ChatGPT looks like it could replace crowdsourced workers for the types of tasks investigated. Given that human moderators complain of the trauma of reviewing toxic content, having AI software take over people’s mundane tasks could benefit their mental health.
Tools such as ChatGPT may be great candidates to replace or reduce human annotation for ethically sensitive tasks.
“This affects unpleasant and demanding annotation tasks, such as hate speech detection, and has a negative psychological impact on human annotators,” said Gilardi. “In other words, tools such as ChatGPT may be the best candidates to replace or reduce human annotation for ethically sensitive tasks performed by humans.”
Also recently there was an opinion that ChatGPT and its ilk have one advantage. It’s all about highlighting the tedious and monotonous tasks you have to do every day, like summarizing reports, emailing your boss, writing boilerplate code, or completing homework. By offering to tackle tedious and repetitive tasks, the model can absorb some of that boredom.
Recent Goldman Sachs Report [PDF] characterize the adoption of generative AI as productivity-enhancing rather than job-destroying, we note. “Extrapolating our estimates globally suggests that generative AI could expose his 300 million worth of full-time jobs to automation,” it said. ®
