As legislative elections in France approach this summer, a research team decided to reach out to hundreds of citizens to interview them about their views on important issues. But the questioner wasn’t a human researcher — it was an AI chatbot.
To prepare ChatGPT to take on this role, the researchers began by motivating the AI bot to behave as it saw professors talking about its training data. The direct message, according to the paper published by the researchers, was: “You are a professor at one of the world’s leading research universities, specializing in high-quality research methods who specialize in conducting interviews. Next, you will conduct an interview with the respondent to find out the motivations and thoughts of the participants regarding their choice to vote during the legislative elections on June 30, 2024, in France, a few days after the interview.”
Human subjects, on the other hand, were told that a chatbot would be conducting an online interview instead of a human, and they were identified for participation using a program called Prolific, which is commonly used by researchers to find research participants.
Part of the project’s research question was whether the participants would be game to share their ideas with the bot, and whether ChatGPT would stay on topic, and, do a professional job enough to solicit useful responses.
The chatbot is part of an experiment by two professors at the London School of Economics, who say AI could change the game when it comes to measuring public opinion in various fields.
“It can really speed up the pace of research,” said Xavier Jaravel, one of the professors leading the research. He noted that AI is already being used in the natural sciences to automate parts of the testing process. For example, this year’s Nobel Prize in chemistry went to scientists who use AI to predict protein folding.
And Jaravel hopes that AI interviewers can allow more researchers in many fields to sample public opinion than is possible and more cost-effective for human interviewers. That could end up creating major changes in academics across the country, adding public opinion and knowledge to the playbook of many academics.
But some researchers doubt that AI bots should replace researchers in the deeply human task of assessing people’s thoughts and feelings.
“It’s a myth to think that having more participants automatically makes the course better — and that’s not necessarily true,” said Andrew Gillen, a teaching assistant professor in the first-year engineering program at Northeastern University. He argues that in most cases, “in-depth conversations with a select group are often more meaningful” – and that those should be done by people.
AI doesn’t judge
In a survey of French voters, as well as another test that used an inquiry method about what makes life meaningful, many participants said in a post-survey that they preferred a chatbot when it came to sharing their opinions on more personal topics.
“Half of the respondents said they would like to take the interview again, or have the AI interview again,” Jaravel said. “And the reason is that they feel that AI is a non-judgmental entity. So that they can share their thoughts freely, and not be judged. And they thought that with a person, they would feel defeated, which is possible. “
About 15% of participants said they would prefer to ask a person, and about 35% said they were not interested in a chatbot or people.
The researchers also provided transcripts of the chatbot conversations to trained sociologists to assess the quality of the conversations, and the experts determined that the AI interviewer was comparable to a “professional interviewer,” Jaravel said. The paper on their research points out, however, that “AI-led conversations are not the same as the best human experts.”
The researchers are encouraged by these findings, and have released their forum for free for any other researcher to try.
Jaravel acknowledges that the in-depth interviews that are so common in ethnographic research are far superior to anything their interview system can do. But he says a chatbot can gather much richer information than the kind of static online survey that’s typical when researchers want to sample a large population. “So we think what we can do with the tool here is really improve that kind of research because you can get a lot of information,” he told EdSurge.
Gillen, the Northeastern researcher, argues that there is something important that no chatbot will be able to do that is important even when conducting surveys — something he calls “position.” An AI chatbot is vulnerable and cannot understand what or why it is asking questions, and that alone will change the answers, he argues. “You’re changing the intervention to a bot and not a human,” he adds.
Gillen says that when she was interviewing for a faculty position, the college asked her to videotape her responses to a series of set questions, in what was called a “one-way interview.” And you say you found the format different.
“It’s basically the same” as answering questions on a Zoom call with people, he says, “but it still felt awful.” While that experience doesn’t involve AI, he says he thinks the chatbot he’s talking to would feel the same way.
Bring the Words
For Jaravel, however, the hope is that this approach can help fields that don’t currently ask for public inclusion to start doing so.
“In economics we rarely talk to people,” he said, noting that researchers in the field often look to large datasets of economic indicators as an important source of research.
The next step for the researchers is to try to add voice capabilities to their platform, so that the bot can ask questions verbally instead of a text conversation.
What does a survey involving French voters reveal?
Based on chatbot interviews with 422 French voters, the researchers found that participants focus on very different issues according to their political leanings. “Respondents on the left are driven by a desire to reduce inequality and promote green reform through various policies,” the researchers concluded in their paper. “On the contrary, respondents at the center highlighted the importance of ensuring the continuity of progressive policies and economic stability, i.e. preserving the agenda and legacy of the President. Finally, right-wing voters highlight immigration (77 percent), insecurity and crime (47 percent) and policies that favor French citizens over immigrants (30 percent) as their main reasons for support.”
The researchers argue that the findings “shed new light on these questions, showing that our
a simple tool can be deployed very quickly to investigate changes in the political environment in real time. “
This article was originally published on EdSurge. EdSurge is a non-profit newsroom that combines education in real journalism with research. Sign up for them newsletters. And follow their own Complete AI installation.
Source link
