With generative AI getting all the hype, a new study from MIT researchers sheds light on the technology’s impact on jobs, including cover letter writing, sensitive emails and cost-benefit analysis. workers assigned tasks were found to be more productive.
The research tasks were not exact replications of real work. That is, precise factual accuracy and context regarding company goals, customer preferences, etc. were not required. Still, many of the survey participants said the assignments were similar to what they wrote on the job, and that the benefits were significant. Access to the assistive chatbot ChatGPT reduced the time it took for the worker to complete a task by 40% and improved his output quality by 18% as measured by an independent evaluator.
The researchers hope to publish the study in open access in a journal today. chemistryto help people understand the impact of AI tools like ChatGPT on their workforce.
“What we can say for sure is that generative AI will have a big impact on white-collar jobs,” said fellow Ph.D. says PhD student Shaked Noy. “I think what our research shows is that this kind of technology has important applications in white-collar jobs. It’s useful technology. But is it a good thing?” It’s still too early to tell exactly if it’s a bad thing or how society will adapt.”
Simulate chatbot work
For centuries, people have feared that new technological advances would lead to mass automation and job losses. But new technologies also create new jobs, and increasing worker productivity can have a net positive effect on the economy.
“When economists think about new technological developments, productivity is the number one priority,” Neu said. “The classic idea of economics is that the most important thing that advances in technology do is to increase productivity in the sense that it enables us to produce economic output more efficiently. .”
To study the impact of generative AI on worker productivity, the researchers recruited 453 college-educated marketers, grant makers, consultants, data analysts, HR professionals and managers. , gave two writing tasks specific to each profession. 20-30 minute tasks include writing cover letters for grant applications, emails about reorganizations, and analytical plans to help companies decide which customers to send push notifications to based on specific customer data. will be An experienced professional in the same profession as each participant rated each submission as if it had been encountered on the job. The evaluator did not know which submissions were made using her ChatGPT.
In the second challenge, half of the participants were given access to the chatbot ChatGPT-3.5 developed by OpenAI. These users completed his task 11 minutes faster than the control group, and his average quality rating improved by 18%.
The data also showed that performance inequality among workers decreased. This means that the worker who received a lower rating on the first task benefited more from using her ChatGPT on the second task.
The researchers noted that while these tasks are largely representative of what professionals see in their work, they have many limitations. We used anonymous participants, which prevented researchers from requesting contextual knowledge about specific companies or customers. Also, while the real-world tasks might be more open-ended, we needed to give clear instructions for each assignment. Additionally, researchers believed it was impractical to hire a fact-checker to assess the accuracy of the output. Accuracy is a big issue for today’s generative AI technology.
The researchers said these limitations could reduce ChatGPT’s real-world productivity potential. Still, they believe the results show the promise of this technology. This idea is supported by another study. Workers exposed to ChatGPT during the experiment were twice as likely to report using ChatGPT at their actual job two weeks after the experiment.
“The experiments demonstrated that even though the real-world speed advantage is small, it provides a significant speed advantage because of the need to spend time fact-checking and creating prompts,” Neu said. increase.
get macro view
In this study, we explored the impact of tools like ChatGPT on specific writing tasks. But estimating the impact of generative AI to understand its impact on the economy is more difficult. That’s what the researchers hope to tackle next.
“There are many other factors that influence wages, employment, and cross-sectoral shifts, for which we need evidence that is not in our paper,” Chan says. “But the time savings and quality improvements are huge for our papers, and at least for certain types of work this seems pretty revolutionary.”
Both researchers believe that even if ChatGPT is found to improve productivity for many workers, much work remains to figure out how society should respond to the spread of generative AI. I agree that it is.
“The policies needed to adapt to these technologies could be very different depending on what further research reveals,” Chan says. “If you think this will push up wages for low-wage workers, it makes a lot more sense than pushing up wages for already high-income earners, widening wage inequality. I think there are a lot of emotional and political implications.”
This work was supported by an Emergent Ventures grant, the George Mason University Mercatas Center, a George and Obie Schultz Endowment grant, an MIT Department of Economics, and a National Science Foundation Graduate Research Fellowship grant.
