new paper According to researchers at the Swiss university EPFL, between 33% and 46% of distributed cloud workers using Amazon’s Mechanical Turk service “cheated” when performing specific tasks assigned to them. . work. If this kind of behavior becomes widespread, it can become a pretty serious problem.
Amazon’s Mechanical Turk has long been a haven for frustrated developers who want humans to do their jobs. In a nutshell, it’s an application programming interface (API) that provides tasks to humans, lets humans perform tasks, and returns results. These tasks are usually the kind that you would like your computer to do better. According to Amazon, examples of such tasks are: “Drawing bounding boxes to build high-quality datasets for computer vision models. This task is too vague for a purely mechanical solution, and potentially too voluminous even for a large team of human experts.” there is. “
A data scientist treats a dataset differently depending on its origin, whether it is human-generated or generated by a large-scale language model (LLM). But Mechanical Turk’s problem is more serious than you think. Choosing to use Mechanical Turk rather than a machine-generated solution because AI is available cheaply enough right now, his manager relies on humans being better than robots. Polluting that large amount of data can have serious repercussions.
“Distinguishing between LLM and human-generated text is a challenge for machine learning models and humans alike,” said the researchers. So researchers created a methodology to figure out whether text-based content was created by humans or by machines.
The test involved asking crowdsourced employees to summarize research abstracts from the New England Journal of Medicine into 100-word abstracts.this is worth noting accurately It’s the kind of task that generative AI technologies like ChatGPT are good at.
