In a recent article published in a magazine chemistryUniversity of Pennsylvania, University of Waterloo, University of Toronto, and Yale University investigated how artificial intelligence, especially large-scale language models (LLMs), can change the nature of work.
“LLM may replace human participants in data collection,” says Professor Philip Tetlock of Penn Integrates Knowledge University, who co-authored the article.
“In fact, LLM has already demonstrated its ability to generate realistic survey responses about consumer behavior. LLM will revolutionize human-based forecasting in the next three years. But it doesn’t make sense to try to make probabilistic judgments in a serious policy debate, I’d say 90% of the time.How humans react to all this is another matter, of course.”
Traditionally, the social sciences have relied on various methods such as questionnaires, behavioral tests, observational studies, and experiments. A common goal in social science research is to obtain generalized representations of the characteristics of individuals, groups and cultures and their dynamics. The emergence of advanced AI systems may change the landscape of data collection in the social sciences.
“What we wanted to explore in this article is how research practices in the social sciences can be adapted and even reinvented to harness the power of AI,” says University of Waterloo Psychology Associate Professor Igor Grossman, lead author of .
Tetlock et al. point out that artificial intelligence models, especially large-scale language models trained on vast amounts of text data, are increasingly capable of simulating human-like reactions and behaviors. are doing. This provides new opportunities for rapidly testing theories and hypotheses about human behavior at scale.
“Because AI models can represent vast amounts of human experience and perspectives, they may be able to generate a wider variety of responses with greater degrees of freedom than traditional human participation methods, alleviating generalizability concerns in research. It helps us do that,” says Grossman.
Although opinions about the feasibility of this application of advanced AI systems vary, studies using simulated participants could be used to generate new hypotheses that can be confirmed in human populations. I have.
However, researchers caution about some pitfalls in this approach, including the fact that LLMs are often trained to eliminate sociocultural biases that exist in real people. This means that sociologists who use AI in this way cannot study those biases.
The researchers point out that researchers need to establish guidelines for LLM governance in research.
“The real concerns about data quality, fairness, and fair access to powerful AI systems will be substantial,” said University of Waterloo professor Dawn Parker, co-author of the article. To tell. “Thus, like all scientific models, we need to ensure that the Social Science LLM is open source. This means that its algorithms and ideally the data can be scrutinized, tested and modified by anyone.” Only by maintaining transparency and reproducibility can we ensure that AI-powered social science research truly contributes to our understanding of the human experience.”
Philip Tetlock is Professor of Democracy and Civil Rights at the Leonor Annenberg Penn Integrates Knowledge College, with appointments at the School of Arts and Sciences and the Wharton School.
This work was supported by the Canadian Social Sciences and Humanities Research Council (611-2020-0190 and 435-2014-0685) and the John Templeton Foundation (62260).
