Can language models stand in for human belief?

Researchers and companies increasingly use large language models to simulate people, to answer surveys, judge arguments or fact-check. We test whether that works by replicating established human experiments with LLMs, and we study how personalized AI changes what people, and whole social networks, believe.

Initial beliefs of 12 language models compared with people's (dashed line): no model matches the human distribution.
12language models tested against a classic study of how beliefs change in social networks
1,265participants in our pre-registered experiment around the 2024 US presidential election
3chatbots tested at fact-checking political misinformation: ChatGPT, Gemini and Claude

2025

Can chatbots fact-check?

We asked ChatGPT, Gemini and Claude to find credible sources for political claims and then write persuasive corrections. The models struggled to ground their answers in real news sources and cited left-leaning sources more often, a warning against relying on prompting alone for AI fact-checking. Personalizing the responses to a reader's background made them more varied and persuasive.

  • How LLMs Fail to Support Fact-Checking A. M. Proma, N. Pate, J. Druckman, G. Ghoshal, H. He, and E. Hoque arXiv:2503.01902, 2025 We tested ChatGPT, Gemini and Claude at countering political misinformation and found consistent gaps, especially in grounding their answers in reliable sources.
  • Personalizing LLM Responses to Combat Political Misinformation A. M. Proma, N. Pate, J. N. Druckman, G. Ghoshal, and E. Hoque ACM Conference on User Modeling, Adaptation and Personalization (UMAP 2025), 2025 Can a chatbot correct political misinformation better if it tailors its answer to the reader? This study tests personalized LLM responses as a counter to misinformation.

2025

Personalized AI in social networks

In a pre-registered experiment with 1,265 people around the 2024 US presidential election, having a personalized LLM in one's network moved people's beliefs toward the truth. People chose to follow the LLM and then formed new connections with others whose beliefs resembled its more accurate ones.

2026

LLMs are not human proxies

We replicated four political motivated-reasoning studies with base LLMs, and an established study of belief dynamics in social networks with 12 LLMs. In both, the models did not behave like people: they were more conformist, did not reproduce people's starting beliefs, and handled motivation and similarity between people differently. Researchers should be careful using LLMs in place of human participants.

  • Replicating Human Motivated Reasoning Studies with LLMs N. Pate, A. M. Proma, H. He, J. N. Druckman, D. C. Molden, G. Ghoshal, and E. Hoque arXiv:2601.16130, 2026 People often reason toward the conclusion they want. We replicated classic motivated-reasoning experiments with base language models to test whether they show the same bias, which matters for using LLMs to simulate survey respondents.
  • Can LLMs Emulate Human Belief Dynamics? A. M. Proma, N. Pate, J. N. Druckman, G. Ghoshal, H. He, and E. Hoque arXiv:2605.18781, 2026 We replicated an established study of how beliefs form and change in social networks with 12 language models. The answer is a clear no: the models fail in systematic ways, a caution for using LLMs as stand-ins for people.

People

Led by PhD students Adiba Mahbub Proma (Rising Stars in EECS 2026-27) and Neeley Pate, with political scientist James Druckman, physicist Gourab Ghoshal, psychologist Daniel Molden and computer scientist Hangfeng He.

More

← All research