Complement Consistency in LLM-Agent Opinion Simulations
What it is
This paper investigates whether Large Language Models (LLMs) maintain logical consistency when simulating agent opinions, specifically testing if reporting the probability of a claim and its complement sums to one. It evaluates three small open models (Qwen3-4B, Llama-3.2-3B, Gemma-3-4B) on simulated network states and real fact-checking claims, finding that models systematically fail this consistency check, with paired reports summing to values significantly greater than one.
The problem it solves
LLM-agent opinion simulations often treat reported probabilities as true beliefs, but models fail to maintain logical consistency between a claim and its complement, leading to biased and unreliable simulation dynamics.
What you could use it for
- Validating the reliability of LLM-based social simulation outputs
- Designing evaluation benchmarks for probabilistic reasoning in LLM agents
- Developing correction methods for inconsistent probability estimates in multi-agent systems
Why it matters
If the findings hold, it suggests that current LLMs cannot be trusted to provide calibrated probabilistic beliefs in agent simulations without explicit consistency enforcement, potentially invalidating many existing studies that rely on raw LLM probability outputs as ground truth for agent states.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.