At first glance, recent news stories may seem to spell doom for the polling industry.
Instead of asking real people for their opinions, some companies are asking artificial intelligence for their opinions. will do think. In other cases, bad actors are using AI to fake survey responses at scale. Some polls face serious data quality issues due to fake respondents who do not truthfully complete the survey.
For people outside the polling industry, it can be difficult to understand what’s going on and which polls to trust. In this Q&A, Courtney Kennedy, Vice President of Methodology and Innovation at Pew Research Center, answers frequently asked questions about the current state of U.S. polling.
Some companies are now asking AI what the public thinks, rather than asking the people themselves. This is sometimes called “silicon sampling.” Is the Pew Research Center doing that?
No, we only interview real people. We do not use AI to inform public opinion. There are ethical and scientific concerns about using AI in place of humans in public opinion polls.
What do you think are the main issues with silicon sampling?
Opinion polls are fundamentally about people, what they think and what they experience. Opinion polls give the public a voice in politics, business, and other areas. In politics, it lets leaders know what hardships people are experiencing and what improvements they would like to see from their government. If we stop polling people and just assume that AI knows the answers, we risk misunderstanding what’s really going on in society.
There are also scientific concerns. There is a lot of research being done on how AI can work in place of human interviews. We conducted some of this research ourselves, but only for experimental learning purposes, not for reporting purposes.
These studies found that AI estimates tend to fixate on groups of people, have a harder time representing Republican views than Democratic views, and tend to underestimate the level of public disagreement.
Over time, AI may even more accurately mimic the answers that real humans would give. Still, the key philosophical point of polls is that they ask real people their opinions.
In addition to silicon sampling, malicious actors Use AI to fake survey responses On a scale. Does that matter to the Pew Research Center?
No, that threat also applies to “opt-in” surveys. These are surveys that people can actively sign up and participate in. For example, responding to social media advertisements that offer rewards for completing surveys. Because it is easy to adopt fake identities online, opt-in voting opens the door to AI and bad actors looking to commit fraud.
Pew Research Center does not use opt-in surveys. Use probability-based sampling instead. In other words, choose real, real people, not online. We start with a huge list of all home addresses in the US and randomly select a few of them. We first contact people by post and invite only a carefully selected sample of the public to participate in the survey each year. Your chances of being selected are slim and you cannot register or nominate yourself to complete the survey. This means that malicious actors do not have the ability to volunteer on our panels.
Can’t people who take Pew Research Center polls use AI to answer their questions?
It’s possible, but scale matters. Because anyone can sign up for opt-in surveys, bad actors can create multiple fake accounts and complete dozens or even hundreds of surveys each day to maximize their financial rewards.
Let’s say someone creates five AI bot accounts online and uses them to complete 200 surveys in a day, earning them $1 per survey. That person could hypothetically earn $30,000 a month by using AI to quickly complete large numbers of opt-in surveys.
Using probability panels like the one we use at Pew Research Center, such large-scale fraud is impossible. You cannot create multiple accounts or answer surveys throughout the day. Each of our respondents has one account, completes an average of less than two surveys per month, and receives an average of $11 per survey. People who fill out our surveys using AI could hypothetically earn $22 a month, which is not a hefty salary for a fraudster.
Would someone trying to fool pollsters want to make $30,000 a month, or $22 a month?
What about fake respondents? What are they and what threat do they pose to the polls?
Fake respondents are survey participants who make no effort to answer the questions honestly and instead try to complete the survey as quickly as possible in order to receive a monetary reward. One characteristic of fake respondents is that they tend to give affirmative answers such as “yes” or “approve.” This trend of fake respondents leads to false conclusions and requires news organizations to retract articles based on opt-in polls.
The fundamental reason fake respondents exist is that opt-in surveys typically encourage anyone interested to sign up. Rigorous research is the opposite: researchers carefully recruit people. Users cannot sign up on their own.
Are opt-in surveys always problematic?
no. In some cases, opt-in voting can produce results similar to probability-based voting. Opt-in polls and probability-based polls can yield similar results if polls are used only to measure the opinions of all adults on a particular topic, such as support or disapproval of the president.
However, opt-in polls have been shown to generate spurious data on young adults and estimates of relatively rare behaviors. These rare behaviors include belief in conspiracy theories, devotion to the Orthodox Christian church, military service, and support for political violence.
So are probability polls always reliable?
Not necessarily. Assessing how respondents were recruited is important, but it’s not the only thing that matters. For example, in recent elections, probability-based polls were not properly weighted, resulting in wildly off results.
To conduct a reliable poll, it must be carefully designed from start to finish. Using probability-based samples is the best start, but other practices are important as well.
Probability polls, like those conducted by Pew Research Center, tend to be more expensive than opt-in polls. why is that?
Our research is more expensive because it takes time and effort to obtain participation from a randomly selected sample of Americans. We recruit people offline and in the real world through letters mailed to their home addresses. We use random sampling to ensure that nearly all U.S. adults have the potential to be selected for the study. And we’re allowing people to answer questions over the web or by phone (because research shows some groups of Americans are reluctant to take surveys online). All the effort to be rigorous costs money.
